[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:49:38 dist-test-slave-4tdj krb5kdc[32125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:49:38 dist-test-slave-4tdj krb5kdc[32125](info): set up 2 sockets
Aug 09 04:49:38 dist-test-slave-4tdj krb5kdc[32125](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 04:49:39.457211 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.007s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:49:39 dist-test-slave-4tdj krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:49:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:49:39.616365 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:33503
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:33503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:49:40.120584 32144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:49:40.121007 32144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:49:40.121126 32144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:49:40.130045 32144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:49:40.130174 32144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:49:40.130242 32144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:49:40.130327 32144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:49:40.130450 32144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:49:40.142256 32144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:33503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:33503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:49:40.145426 32144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:49:40.147161 32144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:49:40.169374 32151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:40.169545 32153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:40.170765 32155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:49:40.170966 32144 server_base.cc:1034] running on GCE node
I20260809 04:49:40.172109 32144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:49:40.174304 32144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:49:40.175529 32144 hybrid_clock.cc:648] HybridClock initialized: now 1786250980175451 us; error 94 us; skew 500 ppm
I20260809 04:49:40.176026 32144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:49:40 dist-test-slave-4tdj krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250980, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:49:40.179880 32144 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:49:40.182180 32144 webserver.cc:460] Webserver started at http://127.31.93.62:46175/ using document root <none> and password file <none>
I20260809 04:49:40.183322 32144 fs_manager.cc:359] Metadata directory not provided
I20260809 04:49:40.183480 32144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:49:40.183976 32144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:49:40.189718 32144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "de818406d7034df6b022a024e2057149"
format_stamp: "Formatted at 2026-08-09 04:49:40 on dist-test-slave-4tdj"
server_key: "777d159c4d16c5453d3e7b3902e942cd"
server_key_iv: "85d26c01c4fc0e30fb55ff4178ec4433"
server_key_version: "encryptionkey@0"
I20260809 04:49:40.190621 32144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "de818406d7034df6b022a024e2057149"
format_stamp: "Formatted at 2026-08-09 04:49:40 on dist-test-slave-4tdj"
server_key: "777d159c4d16c5453d3e7b3902e942cd"
server_key_iv: "85d26c01c4fc0e30fb55ff4178ec4433"
server_key_version: "encryptionkey@0"
I20260809 04:49:40.198815 32144 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260809 04:49:40.206918 32168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:40.208825 32144 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260809 04:49:40.209100 32144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "de818406d7034df6b022a024e2057149"
format_stamp: "Formatted at 2026-08-09 04:49:40 on dist-test-slave-4tdj"
server_key: "777d159c4d16c5453d3e7b3902e942cd"
server_key_iv: "85d26c01c4fc0e30fb55ff4178ec4433"
server_key_version: "encryptionkey@0"
I20260809 04:49:40.209296 32144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:49:40.228502 32144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:49:40.234275 32144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:49:40.234712 32144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:49:40.268355 32144 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:33503
I20260809 04:49:40.268464 32257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:33503 every 8 connection(s)
I20260809 04:49:40.271787 32144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:49:40.280181 32259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:40.282892 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 32144
I20260809 04:49:40.283339 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:49:40.285797 32116 external_mini_cluster.cc:1428] Setting key 5d573fb6673cef6f1714511328c368e7
I20260809 04:49:40.299595 32259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149: Bootstrap starting.
I20260809 04:49:40.303978 32259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:40.306736 32259 log.cc:824] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149: Log is configured to *not* fsync() on all Append() calls
I20260809 04:49:40.312031 32259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149: No bootstrap required, opened a new log
Aug 09 04:49:40 dist-test-slave-4tdj krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:40.327271 32259 raft_consensus.cc:348] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de818406d7034df6b022a024e2057149" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33503 } }
I20260809 04:49:40.327796 32259 raft_consensus.cc:374] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:40.327994 32259 raft_consensus.cc:729] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de818406d7034df6b022a024e2057149, State: Initialized, Role: FOLLOWER
I20260809 04:49:40.329419 32259 consensus_queue.cc:260] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de818406d7034df6b022a024e2057149" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33503 } }
I20260809 04:49:40.329843 32259 raft_consensus.cc:388] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:49:40.330248 32259 raft_consensus.cc:482] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:49:40.330616 32259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:40.333982 32259 raft_consensus.cc:504] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de818406d7034df6b022a024e2057149" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33503 } }
I20260809 04:49:40.335237 32259 leader_election.cc:302] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de818406d7034df6b022a024e2057149; no voters:
I20260809 04:49:40.336382 32259 leader_election.cc:288] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:49:40.336472 32266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:49:40.337951 32263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:40.292646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49748 (local address 127.31.93.62:33503)
0809 04:49:40.293935 (+ 1289us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:40.293954 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:40.295164 (+ 1210us) server_negotiation.cc:407] Connection header received
0809 04:49:40.306569 (+ 11405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:40.306588 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:40.307141 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:40.307547 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:40.312154 (+ 4607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:40.313858 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:40.316672 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:40.317704 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:40.317902 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:40.326181 (+ 8279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:40.326234 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:40.326271 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:40.326339 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:40.331010 (+ 4671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:40.331795 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:40.331804 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:40.331812 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:40.332059 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:40.332526 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:40.332534 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:40.332537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:40.333427 (+ 890us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:40.333899 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:40.334748 (+ 849us) server_negotiation.cc:299] Negotiation successful
0809 04:49:40.335176 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"thread_start_us":479,"threads_started":1}
I20260809 04:49:40.338021 32266 raft_consensus.cc:686] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [term 1 LEADER]: Becoming Leader. State: Replica: de818406d7034df6b022a024e2057149, State: Running, Role: LEADER
I20260809 04:49:40.338846 32266 consensus_queue.cc:237] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de818406d7034df6b022a024e2057149" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33503 } }
I20260809 04:49:40.341444 32259 sys_catalog.cc:564] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:49:40.352094 32269 sys_catalog.cc:455] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de818406d7034df6b022a024e2057149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de818406d7034df6b022a024e2057149" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33503 } } }
I20260809 04:49:40.352137 32270 sys_catalog.cc:455] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de818406d7034df6b022a024e2057149. Latest consensus state: current_term: 1 leader_uuid: "de818406d7034df6b022a024e2057149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de818406d7034df6b022a024e2057149" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33503 } } }
I20260809 04:49:40.352741 32269 sys_catalog.cc:458] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [sys.catalog]: This master's current role is: LEADER
I20260809 04:49:40.352866 32270 sys_catalog.cc:458] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149 [sys.catalog]: This master's current role is: LEADER
I20260809 04:49:40.355335 32281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:49:40.368225 32281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:49:40.380748 32281 catalog_manager.cc:1347] Generated new cluster ID: 16dfa842aa334aec89b678bb9c82ca90
I20260809 04:49:40.380863 32281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:49:40.403426 32281 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:49:40.405750 32281 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:49:40.418686 32281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de818406d7034df6b022a024e2057149: Generated new TSK 0
I20260809 04:49:40.419948 32281 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:49:40.477092 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33503
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:49:40.907594 32294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:49:40.907927 32294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:49:40.908113 32294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:49:40.916431 32294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:49:40.916550 32294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:49:40.916697 32294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:49:40.928048 32294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:49:40.929425 32294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:49:40.930999 32294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:49:40.946352 32308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:40.947018 32310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:40.947119 32313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:49:40.948169 32294 server_base.cc:1034] running on GCE node
I20260809 04:49:40.948812 32294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:49:40.949824 32294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:49:40.951035 32294 hybrid_clock.cc:648] HybridClock initialized: now 1786250980950993 us; error 31 us; skew 500 ppm
I20260809 04:49:40.951344 32294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:49:40 dist-test-slave-4tdj krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250980, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:49:40.955519 32294 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:49:40.957288 32294 webserver.cc:460] Webserver started at http://127.31.93.1:45501/ using document root <none> and password file <none>
I20260809 04:49:40.957957 32294 fs_manager.cc:359] Metadata directory not provided
I20260809 04:49:40.958140 32294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:49:40.958500 32294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:49:40.961999 32294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "12f7dc3ea1b14f2fbf9933627954663f"
format_stamp: "Formatted at 2026-08-09 04:49:40 on dist-test-slave-4tdj"
server_key: "6e2afa115c61dfe62a3ca45aebd5b847"
server_key_iv: "b9c8c4d7458fe382497d9f86103fcaaf"
server_key_version: "encryptionkey@0"
I20260809 04:49:40.962932 32294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "12f7dc3ea1b14f2fbf9933627954663f"
format_stamp: "Formatted at 2026-08-09 04:49:40 on dist-test-slave-4tdj"
server_key: "6e2afa115c61dfe62a3ca45aebd5b847"
server_key_iv: "b9c8c4d7458fe382497d9f86103fcaaf"
server_key_version: "encryptionkey@0"
I20260809 04:49:40.968802 32294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 04:49:40.973143 32325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:40.974563 32294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 04:49:40.974733 32294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "12f7dc3ea1b14f2fbf9933627954663f"
format_stamp: "Formatted at 2026-08-09 04:49:40 on dist-test-slave-4tdj"
server_key: "6e2afa115c61dfe62a3ca45aebd5b847"
server_key_iv: "b9c8c4d7458fe382497d9f86103fcaaf"
server_key_version: "encryptionkey@0"
I20260809 04:49:40.974928 32294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:49:40.995074 32294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:49:40.999835 32294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:49:41.000164 32294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:49:41.001467 32294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:49:41.003535 32294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:49:41.003609 32294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:41.003777 32294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:49:41.003845 32294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:41.064211 32294 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:43053
I20260809 04:49:41.064301 32539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:43053 every 8 connection(s)
I20260809 04:49:41.065627 32294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:49:41.074894 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 32294
I20260809 04:49:41.075111 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:49:41.075784 32116 external_mini_cluster.cc:1428] Setting key 4400d03b764bf5cc00168e70c1ff926d
Aug 09 04:49:41 dist-test-slave-4tdj krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250980, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:41.089788 32546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:41.069495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:54777 (local address 127.31.93.62:33503)
0809 04:49:41.070003 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:41.070008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:41.071412 (+ 1404us) server_negotiation.cc:407] Connection header received
0809 04:49:41.073706 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:41.073712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:41.073790 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:41.073887 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:41.075744 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:41.076771 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:41.078495 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:41.078976 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:41.079086 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:41.083204 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:41.083209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:41.083213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:41.083244 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:41.086702 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:41.087399 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:41.087405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:41.087408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:41.087588 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:41.087956 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:41.087960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:41.087963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:41.088277 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:41.088408 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:41.089094 (+ 686us) server_negotiation.cc:299] Negotiation successful
0809 04:49:41.089302 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":205,"threads_started":1}
I20260809 04:49:41.090550 32547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:41.070145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:33503 (local address 127.31.93.1:54777)
0809 04:49:41.071255 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:41.071293 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:41.073435 (+ 2142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:41.074181 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:41.074190 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:41.074848 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:41.075578 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:41.075596 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:41.076917 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:41.076921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:41.078365 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:41.078377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:41.079217 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:41.079226 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:41.079522 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:41.080193 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:41.080216 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:41.082943 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:41.086838 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:41.086848 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:41.086862 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:41.087265 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:41.087680 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:41.087683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:41.087685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:41.087850 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:41.088385 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:41.088391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:41.088677 (+ 286us) client_negotiation.cc:769] Sending connection context
0809 04:49:41.088950 (+ 273us) client_negotiation.cc:240] Negotiation successful
0809 04:49:41.089315 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":642,"threads_started":1}
I20260809 04:49:41.092236 32540 heartbeater.cc:347] Connected to a master server at 127.31.93.62:33503
I20260809 04:49:41.092707 32540 heartbeater.cc:464] Registering TS with master...
I20260809 04:49:41.093710 32540 heartbeater.cc:511] Master 127.31.93.62:33503 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
I20260809 04:49:41.098541 32201 ts_manager.cc:194] Registered new tserver with Master: 12f7dc3ea1b14f2fbf9933627954663f (127.31.93.1:43053)
I20260809 04:49:41.101945 32201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:54777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:49:41.119374 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33503
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:49:41.615793 32553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:49:41.616086 32553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:49:41.616284 32553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:49:41.624729 32553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:49:41.624939 32553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:49:41.625089 32553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:49:41.636142 32553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:49:41.637243 32553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:49:41.638473 32553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:49:41.652892 32561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:41.653069 32562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:41.653075 32567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:49:41.654412 32553 server_base.cc:1034] running on GCE node
I20260809 04:49:41.654978 32553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:49:41.655879 32553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:49:41.657135 32553 hybrid_clock.cc:648] HybridClock initialized: now 1786250981657027 us; error 38 us; skew 500 ppm
I20260809 04:49:41.657430 32553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:49:41 dist-test-slave-4tdj krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250981, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:49:41.661245 32553 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:49:41.662916 32553 webserver.cc:460] Webserver started at http://127.31.93.2:37275/ using document root <none> and password file <none>
I20260809 04:49:41.663551 32553 fs_manager.cc:359] Metadata directory not provided
I20260809 04:49:41.663702 32553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:49:41.664053 32553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:49:41.666913 32553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "86891745c8074099bd697d68de89e4bd"
format_stamp: "Formatted at 2026-08-09 04:49:41 on dist-test-slave-4tdj"
server_key: "cb7c7ec34fb19c93b7243fa3396250d7"
server_key_iv: "36b2e4dbe0e03da9742211075fd2a06a"
server_key_version: "encryptionkey@0"
I20260809 04:49:41.667672 32553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "86891745c8074099bd697d68de89e4bd"
format_stamp: "Formatted at 2026-08-09 04:49:41 on dist-test-slave-4tdj"
server_key: "cb7c7ec34fb19c93b7243fa3396250d7"
server_key_iv: "36b2e4dbe0e03da9742211075fd2a06a"
server_key_version: "encryptionkey@0"
I20260809 04:49:41.673207 32553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 04:49:41.676955 32583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:41.678191 32553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:49:41.678359 32553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "86891745c8074099bd697d68de89e4bd"
format_stamp: "Formatted at 2026-08-09 04:49:41 on dist-test-slave-4tdj"
server_key: "cb7c7ec34fb19c93b7243fa3396250d7"
server_key_iv: "36b2e4dbe0e03da9742211075fd2a06a"
server_key_version: "encryptionkey@0"
I20260809 04:49:41.678591 32553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:49:41.694082 32553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:49:41.699158 32553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:49:41.699555 32553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:49:41.701027 32553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:49:41.702961 32553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:49:41.703109 32553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:41.703241 32553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:49:41.703384 32553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:41.776822 32553 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:33057
I20260809 04:49:41.777143 305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:33057 every 8 connection(s)
I20260809 04:49:41.778366 32553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:49:41.787266 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 32553
I20260809 04:49:41.787515 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:49:41.787942 32116 external_mini_cluster.cc:1428] Setting key e15654e9659bb6b99d0e158913487afd
Aug 09 04:49:41 dist-test-slave-4tdj krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250981, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:41.804416 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:41.781636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:46461 (local address 127.31.93.62:33503)
0809 04:49:41.782531 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:41.782538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:41.784004 (+ 1466us) server_negotiation.cc:407] Connection header received
0809 04:49:41.786193 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:41.786199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:41.786281 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:41.786408 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:41.788409 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:41.789676 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:41.791643 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:41.792175 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:41.792290 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:41.797316 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:41.797321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:41.797324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:41.797357 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:41.800687 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:41.801542 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:41.801545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:41.801548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:41.801724 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:41.802351 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:41.802354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:41.802356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:41.802718 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:41.802852 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:41.803593 (+ 741us) server_negotiation.cc:299] Negotiation successful
0809 04:49:41.803816 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":530,"threads_started":1}
I20260809 04:49:41.805675 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:41.782593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:33503 (local address 127.31.93.2:46461)
0809 04:49:41.783833 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:41.783881 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:41.785887 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:41.786670 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:41.786682 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:41.787449 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:41.788237 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:41.788259 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:41.789814 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:41.789818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:41.791501 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:41.791513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:41.792445 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:41.792454 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:41.792783 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:41.793567 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:41.793588 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:41.797046 (+ 3458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:41.800880 (+ 3834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:41.800889 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:41.800901 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:41.801417 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:41.801868 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:41.801873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:41.801877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:41.802203 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:41.802847 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:41.802855 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:41.803226 (+ 371us) client_negotiation.cc:769] Sending connection context
0809 04:49:41.803474 (+ 248us) client_negotiation.cc:240] Negotiation successful
0809 04:49:41.804110 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":545,"threads_started":1}
I20260809 04:49:41.807431 306 heartbeater.cc:347] Connected to a master server at 127.31.93.62:33503
I20260809 04:49:41.807876 306 heartbeater.cc:464] Registering TS with master...
I20260809 04:49:41.809170 306 heartbeater.cc:511] Master 127.31.93.62:33503 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
I20260809 04:49:41.811080 32201 ts_manager.cc:194] Registered new tserver with Master: 86891745c8074099bd697d68de89e4bd (127.31.93.2:33057)
I20260809 04:49:41.812187 32201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:46461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:49:41.837409 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33503
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:49:42.105796 32540 heartbeater.cc:503] Master 127.31.93.62:33503 was elected leader, sending a full tablet report...
W20260809 04:49:42.312491 312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:49:42.312806 312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:49:42.313024 312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:49:42.321952 312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:49:42.322111 312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:49:42.322348 312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:49:42.333171 312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:49:42.334539 312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:49:42.336056 312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:49:42.351780 324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:42.351807 330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:42.351662 327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:49:42.355854 312 server_base.cc:1034] running on GCE node
I20260809 04:49:42.356419 312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:49:42.357515 312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:49:42.358762 312 hybrid_clock.cc:648] HybridClock initialized: now 1786250982358674 us; error 39 us; skew 500 ppm
I20260809 04:49:42.359103 312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:49:42 dist-test-slave-4tdj krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250982, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:49:42.362862 312 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:49:42.364428 312 webserver.cc:460] Webserver started at http://127.31.93.3:45255/ using document root <none> and password file <none>
I20260809 04:49:42.365068 312 fs_manager.cc:359] Metadata directory not provided
I20260809 04:49:42.365177 312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:49:42.365530 312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:49:42.368418 312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "f333063fc06b487bb7bd3b9c6265b577"
format_stamp: "Formatted at 2026-08-09 04:49:42 on dist-test-slave-4tdj"
server_key: "1f2ce4c3c2c470dc4c457ac6560abd0c"
server_key_iv: "df598336aca8b21dd9f1a55f8128567e"
server_key_version: "encryptionkey@0"
I20260809 04:49:42.369125 312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "f333063fc06b487bb7bd3b9c6265b577"
format_stamp: "Formatted at 2026-08-09 04:49:42 on dist-test-slave-4tdj"
server_key: "1f2ce4c3c2c470dc4c457ac6560abd0c"
server_key_iv: "df598336aca8b21dd9f1a55f8128567e"
server_key_version: "encryptionkey@0"
I20260809 04:49:42.375492 312 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260809 04:49:42.379794 337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:42.381114 312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 04:49:42.381256 312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "f333063fc06b487bb7bd3b9c6265b577"
format_stamp: "Formatted at 2026-08-09 04:49:42 on dist-test-slave-4tdj"
server_key: "1f2ce4c3c2c470dc4c457ac6560abd0c"
server_key_iv: "df598336aca8b21dd9f1a55f8128567e"
server_key_version: "encryptionkey@0"
I20260809 04:49:42.381441 312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:49:42.410794 312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:49:42.416162 312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:49:42.416491 312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:49:42.417902 312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:49:42.419652 312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:49:42.419727 312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:42.419838 312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:49:42.419925 312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:42.475010 312 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:42823
I20260809 04:49:42.475066 530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:42823 every 8 connection(s)
I20260809 04:49:42.476246 312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:49:42.485380 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 312
I20260809 04:49:42.485558 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:49:42.485998 32116 external_mini_cluster.cc:1428] Setting key 3506cee9e8ee5af6666f50ec7c209726
Aug 09 04:49:42 dist-test-slave-4tdj krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250982, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:42.499948 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:42.479529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:32893 (local address 127.31.93.62:33503)
0809 04:49:42.480413 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:42.480418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:42.481426 (+ 1008us) server_negotiation.cc:407] Connection header received
0809 04:49:42.484056 (+ 2630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:42.484062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:42.484178 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:42.484276 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:42.486202 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:42.487125 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:42.489672 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:42.490416 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:42.490532 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:42.494080 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:42.494084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:42.494087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:42.494119 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:42.497079 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:42.498091 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:42.498095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:42.498097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:42.498220 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:42.498592 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:42.498596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:42.498597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:42.498850 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:42.498956 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:42.499503 (+ 547us) server_negotiation.cc:299] Negotiation successful
0809 04:49:42.499651 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":629,"threads_started":1}
I20260809 04:49:42.501014 540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:42.480230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:33503 (local address 127.31.93.3:32893)
0809 04:49:42.481271 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:42.481309 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:42.483760 (+ 2451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:42.484542 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:42.484551 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:42.485234 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:42.485996 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:42.486016 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:42.487287 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:42.487291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:42.489517 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:42.489532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:42.490625 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:42.490630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:42.490824 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:42.491404 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:42.491426 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:42.493908 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:42.497217 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:42.497226 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:42.497240 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:42.497998 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:42.498333 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:42.498337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:42.498339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:42.498512 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:42.498945 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:42.498950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:42.499229 (+ 279us) client_negotiation.cc:769] Sending connection context
0809 04:49:42.499440 (+ 211us) client_negotiation.cc:240] Negotiation successful
0809 04:49:42.499781 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":589,"threads_started":1}
I20260809 04:49:42.502429 533 heartbeater.cc:347] Connected to a master server at 127.31.93.62:33503
I20260809 04:49:42.502785 533 heartbeater.cc:464] Registering TS with master...
I20260809 04:49:42.503854 533 heartbeater.cc:511] Master 127.31.93.62:33503 requested a full tablet report, sending...
I20260809 04:49:42.505406 32201 ts_manager.cc:194] Registered new tserver with Master: f333063fc06b487bb7bd3b9c6265b577 (127.31.93.3:42823)
I20260809 04:49:42.506423 32201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:32893
I20260809 04:49:42.514988 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 04:49:42 dist-test-slave-4tdj krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 04:49:42 dist-test-slave-4tdj krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:49:42.575680 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:42.557204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.31.93.1:43053)
0809 04:49:42.558178 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:42.558185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:42.558216 (+ 31us) server_negotiation.cc:407] Connection header received
0809 04:49:42.558416 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:42.558423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:42.558631 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:42.558783 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:42.560660 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:42.561760 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:42.563388 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:42.564227 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:42.564327 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:42.569048 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:42.569060 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:42.569075 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:42.569165 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:42.572669 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:42.573287 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:42.573294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:42.573296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:42.573421 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:42.573804 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:42.573807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:42.573809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:42.574197 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:42.574500 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:42.574862 (+ 362us) server_negotiation.cc:299] Negotiation successful
0809 04:49:42.575197 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":567,"threads_started":1}
W20260809 04:49:42.577126 32372 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36908
Aug 09 04:49:42 dist-test-slave-4tdj krb5kdc[32125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 04:49:42 dist-test-slave-4tdj krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:49:42.621959 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:42.608071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.31.93.1:43053)
0809 04:49:42.608245 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:42.608252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:42.608666 (+ 414us) server_negotiation.cc:407] Connection header received
0809 04:49:42.608848 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:42.608852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:42.608946 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:42.609028 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:42.610333 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:42.611056 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:42.612246 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:42.612709 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:42.612819 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:42.616433 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:42.616438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:42.616442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:42.616474 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:42.619303 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:42.620077 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:42.620087 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:42.620089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:42.620213 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:42.620605 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:42.620610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:42.620611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:42.620833 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:42.621110 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:42.621417 (+ 307us) server_negotiation.cc:299] Negotiation successful
0809 04:49:42.621624 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 04:49:42.623770 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 32294
I20260809 04:49:42.637444 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:42.624729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.31.93.1:43053)
0809 04:49:42.624862 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:42.624866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:42.625199 (+ 333us) server_negotiation.cc:407] Connection header received
0809 04:49:42.625314 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:42.625317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:42.625366 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:42.625436 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:42.626848 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:42.627558 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:42.629149 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:42.629767 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:42.629856 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:42.632322 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:42.632326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:42.632329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:42.632355 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:42.635345 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:42.635867 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:42.635870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:42.635872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:42.635989 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:42.636319 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:42.636322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:42.636324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:42.636572 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:42.636742 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:42.637008 (+ 266us) server_negotiation.cc:299] Negotiation successful
0809 04:49:42.637155 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 04:49:42.638229 32533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:49:42.725971 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 32294
I20260809 04:49:42.746089 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 32553
Aug 09 04:49:42 dist-test-slave-4tdj krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:49:42.761551 560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:42.746984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58560 (local address 127.31.93.2:33057)
0809 04:49:42.747866 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:42.747873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:42.747899 (+ 26us) server_negotiation.cc:407] Connection header received
0809 04:49:42.747961 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:42.747968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:42.748189 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:42.748316 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:42.749381 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:42.750622 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:42.751882 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:42.752450 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:42.752549 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:42.756290 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:42.756301 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:42.756315 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:42.756355 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:42.758921 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:42.759657 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:42.759664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:42.759667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:42.759779 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:42.760222 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:42.760225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:42.760227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:42.760544 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:42.760769 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:42.761021 (+ 252us) server_negotiation.cc:299] Negotiation successful
0809 04:49:42.761227 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":590,"threads_started":1}
I20260809 04:49:42.762750 300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:49:42.840971 306 heartbeater.cc:503] Master 127.31.93.62:33503 was elected leader, sending a full tablet report...
I20260809 04:49:42.843075 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 32553
I20260809 04:49:42.863500 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 312
Aug 09 04:49:42 dist-test-slave-4tdj krb5kdc[32125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:49:42.879177 562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:42.864436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51068 (local address 127.31.93.3:42823)
0809 04:49:42.865053 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:42.865060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:42.865088 (+ 28us) server_negotiation.cc:407] Connection header received
0809 04:49:42.865144 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:42.865159 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:42.865371 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:42.865514 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:42.866886 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:42.867936 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:42.869207 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:42.869755 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:42.869880 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:42.873750 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:42.873761 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:42.873774 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:42.873809 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:42.876341 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:42.876953 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:42.876961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:42.876963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:42.877083 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:42.877434 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:42.877438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:42.877440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:42.877903 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:42.878247 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:42.878571 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 04:49:42.878849 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":321,"threads_started":1}
I20260809 04:49:42.880394 524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:49:42.960290 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 312
I20260809 04:49:42.979451 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 32144
I20260809 04:49:42.980546 32253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:49:43.082201 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 32144
2026-08-09T04:49:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4854 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:49:43 dist-test-slave-4tdj krb5kdc[568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:49:43 dist-test-slave-4tdj krb5kdc[568](info): set up 2 sockets
Aug 09 04:49:43 dist-test-slave-4tdj krb5kdc[568](info): commencing operation
krb5kdc: starting...
W20260809 04:49:44.271085 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:49:44 dist-test-slave-4tdj krb5kdc[568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:49:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:49:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:49:44.432861 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:34565
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:34565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:49:44.733381 585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:49:44.733640 585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:49:44.733770 585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:49:44.740875 585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:49:44.740978 585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:49:44.741050 585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:49:44.741124 585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:49:44.741189 585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:49:44.751166 585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:34565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:34565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:49:44.752324 585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:49:44.753526 585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:49:44.764323 596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:44.764384 593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:44.764356 594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:49:44.764954 585 server_base.cc:1034] running on GCE node
I20260809 04:49:44.765702 585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:49:44.767010 585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:49:44.768213 585 hybrid_clock.cc:648] HybridClock initialized: now 1786250984768160 us; error 41 us; skew 500 ppm
I20260809 04:49:44.768522 585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:49:44 dist-test-slave-4tdj krb5kdc[568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250984, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:49:44.774147 585 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:49:44.775508 585 webserver.cc:460] Webserver started at http://127.31.93.62:36319/ using document root <none> and password file <none>
I20260809 04:49:44.776146 585 fs_manager.cc:359] Metadata directory not provided
I20260809 04:49:44.776262 585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:49:44.776542 585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:49:44.779389 585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "f7002ae87730403f81216c8a912bc270"
format_stamp: "Formatted at 2026-08-09 04:49:44 on dist-test-slave-4tdj"
server_key: "21fd50e410add186f71ce12c86e9619e"
server_key_iv: "a93a1367f583e5a689605a25a70d246f"
server_key_version: "encryptionkey@0"
I20260809 04:49:44.780084 585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "f7002ae87730403f81216c8a912bc270"
format_stamp: "Formatted at 2026-08-09 04:49:44 on dist-test-slave-4tdj"
server_key: "21fd50e410add186f71ce12c86e9619e"
server_key_iv: "a93a1367f583e5a689605a25a70d246f"
server_key_version: "encryptionkey@0"
I20260809 04:49:44.784855 585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260809 04:49:44.788002 607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:44.789129 585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:49:44.789281 585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "f7002ae87730403f81216c8a912bc270"
format_stamp: "Formatted at 2026-08-09 04:49:44 on dist-test-slave-4tdj"
server_key: "21fd50e410add186f71ce12c86e9619e"
server_key_iv: "a93a1367f583e5a689605a25a70d246f"
server_key_version: "encryptionkey@0"
I20260809 04:49:44.789443 585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:49:44.810196 585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:49:44.814029 585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:49:44.814282 585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:49:44.830797 585 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:34565
I20260809 04:49:44.830863 697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:34565 every 8 connection(s)
I20260809 04:49:44.832093 585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:49:44.834717 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 585
I20260809 04:49:44.834923 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:49:44.835541 32116 external_mini_cluster.cc:1428] Setting key 0bd77ace3a87fbacdd36cb06acc34bb4
I20260809 04:49:44.835629 699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:44.849036 699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270: Bootstrap starting.
Aug 09 04:49:44 dist-test-slave-4tdj krb5kdc[568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:44.852567 699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:44.853863 699 log.cc:824] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270: Log is configured to *not* fsync() on all Append() calls
I20260809 04:49:44.856999 699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270: No bootstrap required, opened a new log
I20260809 04:49:44.858237 702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:44.837346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.31.93.62:34565)
0809 04:49:44.838178 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:44.838193 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:44.838245 (+ 52us) server_negotiation.cc:407] Connection header received
0809 04:49:44.840260 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:44.840286 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:44.840749 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:44.841155 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:44.842451 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:44.844368 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:44.845534 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:44.846590 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:44.846783 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:44.850387 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:44.850398 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:44.850416 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:44.850465 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:44.854063 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:44.854812 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:44.854818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:44.854823 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:44.854992 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:44.855352 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:44.855357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:44.855360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:44.855985 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:44.856260 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:44.856599 (+ 339us) server_negotiation.cc:299] Negotiation successful
0809 04:49:44.856998 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":352,"threads_started":1}
I20260809 04:49:44.864606 699 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7002ae87730403f81216c8a912bc270" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34565 } }
I20260809 04:49:44.864943 699 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:44.865043 699 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7002ae87730403f81216c8a912bc270, State: Initialized, Role: FOLLOWER
I20260809 04:49:44.865561 699 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7002ae87730403f81216c8a912bc270" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34565 } }
I20260809 04:49:44.865770 699 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:49:44.865866 699 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:49:44.865973 699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:44.867879 699 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7002ae87730403f81216c8a912bc270" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34565 } }
I20260809 04:49:44.868300 699 leader_election.cc:302] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7002ae87730403f81216c8a912bc270; no voters:
I20260809 04:49:44.868770 699 leader_election.cc:288] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:49:44.868887 705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:49:44.869947 705 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [term 1 LEADER]: Becoming Leader. State: Replica: f7002ae87730403f81216c8a912bc270, State: Running, Role: LEADER
I20260809 04:49:44.870549 705 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7002ae87730403f81216c8a912bc270" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34565 } }
I20260809 04:49:44.870687 699 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:49:44.873250 706 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7002ae87730403f81216c8a912bc270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7002ae87730403f81216c8a912bc270" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34565 } } }
I20260809 04:49:44.873224 707 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7002ae87730403f81216c8a912bc270. Latest consensus state: current_term: 1 leader_uuid: "f7002ae87730403f81216c8a912bc270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7002ae87730403f81216c8a912bc270" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34565 } } }
I20260809 04:49:44.873705 706 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [sys.catalog]: This master's current role is: LEADER
I20260809 04:49:44.873703 707 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270 [sys.catalog]: This master's current role is: LEADER
I20260809 04:49:44.875524 714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:49:44.879575 714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:49:44.888053 714 catalog_manager.cc:1347] Generated new cluster ID: 3f7855614cd64c7bb51f42427eb69936
I20260809 04:49:44.888162 714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:49:44.910349 714 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:49:44.911537 714 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:49:44.924757 714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7002ae87730403f81216c8a912bc270: Generated new TSK 0
I20260809 04:49:44.925530 714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:49:44.969316 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34565
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:49:45.283188 735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:49:45.283435 735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:49:45.283572 735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:49:45.290478 735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:49:45.290580 735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:49:45.290729 735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:49:45.300005 735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34565
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:49:45.301141 735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:49:45.302376 735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:49:45.314481 744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:45.314450 746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:45.314589 748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:49:45.315836 735 server_base.cc:1034] running on GCE node
I20260809 04:49:45.316334 735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:49:45.317131 735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:49:45.318370 735 hybrid_clock.cc:648] HybridClock initialized: now 1786250985318310 us; error 44 us; skew 500 ppm
I20260809 04:49:45.318640 735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:49:45 dist-test-slave-4tdj krb5kdc[568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250985, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:49:45.321559 735 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:49:45.322934 735 webserver.cc:460] Webserver started at http://127.31.93.1:33509/ using document root <none> and password file <none>
I20260809 04:49:45.323556 735 fs_manager.cc:359] Metadata directory not provided
I20260809 04:49:45.323653 735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:49:45.323899 735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:49:45.326676 735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "5a9abd86d80b41f49d3fd663906ab1f4"
format_stamp: "Formatted at 2026-08-09 04:49:45 on dist-test-slave-4tdj"
server_key: "5af8aeca0bd263ca460a784c842aefa2"
server_key_iv: "9d9449957ea36cad3d9285a338eecbc0"
server_key_version: "encryptionkey@0"
I20260809 04:49:45.327381 735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "5a9abd86d80b41f49d3fd663906ab1f4"
format_stamp: "Formatted at 2026-08-09 04:49:45 on dist-test-slave-4tdj"
server_key: "5af8aeca0bd263ca460a784c842aefa2"
server_key_iv: "9d9449957ea36cad3d9285a338eecbc0"
server_key_version: "encryptionkey@0"
I20260809 04:49:45.332085 735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260809 04:49:45.335230 759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:45.336388 735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260809 04:49:45.336546 735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "5a9abd86d80b41f49d3fd663906ab1f4"
format_stamp: "Formatted at 2026-08-09 04:49:45 on dist-test-slave-4tdj"
server_key: "5af8aeca0bd263ca460a784c842aefa2"
server_key_iv: "9d9449957ea36cad3d9285a338eecbc0"
server_key_version: "encryptionkey@0"
I20260809 04:49:45.336719 735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:49:45.362176 735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:49:45.366055 735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:49:45.366322 735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:49:45.367187 735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:49:45.368383 735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:49:45.368454 735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:45.368589 735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:49:45.368640 735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:45.397928 735 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:33869
I20260809 04:49:45.397985 931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:33869 every 8 connection(s)
I20260809 04:49:45.399147 735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:49:45.404111 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 735
I20260809 04:49:45.404341 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:49:45.405102 32116 external_mini_cluster.cc:1428] Setting key 70d284e021f849e06c205266ae00c588
Aug 09 04:49:45 dist-test-slave-4tdj krb5kdc[568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250985, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:45.419466 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:45.401354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:47849 (local address 127.31.93.62:34565)
0809 04:49:45.401761 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:45.401765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:45.402613 (+ 848us) server_negotiation.cc:407] Connection header received
0809 04:49:45.404607 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:45.404612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:45.404687 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:45.404781 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:45.406533 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:45.407337 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:45.409431 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:45.409807 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:45.409893 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:45.413582 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:45.413588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:45.413593 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:45.413631 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:45.416636 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:45.417283 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:45.417289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:45.417292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:45.417437 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:45.417811 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:45.417815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:45.417818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:45.418132 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:45.418273 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:45.418911 (+ 638us) server_negotiation.cc:299] Negotiation successful
0809 04:49:45.419116 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":220,"threads_started":1}
I20260809 04:49:45.420095 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:45.401771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:34565 (local address 127.31.93.1:47849)
0809 04:49:45.402454 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:45.402495 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:45.404320 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:45.405036 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:45.405048 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:45.405749 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:45.406371 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:45.406395 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:45.407490 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:45.407496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:45.409311 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:45.409325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:45.410014 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:45.410021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:45.410314 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:45.410927 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:45.410959 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:45.413347 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:45.416757 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:45.416763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:45.416773 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:45.417154 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:45.417533 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:45.417535 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:45.417537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:45.417700 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:45.418249 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:45.418254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:45.418559 (+ 305us) client_negotiation.cc:769] Sending connection context
0809 04:49:45.418772 (+ 213us) client_negotiation.cc:240] Negotiation successful
0809 04:49:45.419109 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":268,"threads_started":1}
I20260809 04:49:45.421476 935 heartbeater.cc:347] Connected to a master server at 127.31.93.62:34565
I20260809 04:49:45.421794 935 heartbeater.cc:464] Registering TS with master...
I20260809 04:49:45.422667 935 heartbeater.cc:511] Master 127.31.93.62:34565 requested a full tablet report, sending...
I20260809 04:49:45.424865 634 ts_manager.cc:194] Registered new tserver with Master: 5a9abd86d80b41f49d3fd663906ab1f4 (127.31.93.1:33869)
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
I20260809 04:49:45.426512 634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:47849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:49:45.446883 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34565
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:49:45.752177 950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:49:45.752418 950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:49:45.752517 950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:49:45.759414 950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:49:45.759508 950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:49:45.759624 950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:49:45.768860 950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34565
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:49:45.769953 950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:49:45.771245 950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:49:45.783324 958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:45.783249 960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:45.783288 957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:49:45.784194 950 server_base.cc:1034] running on GCE node
I20260809 04:49:45.784741 950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:49:45.785588 950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:49:45.786774 950 hybrid_clock.cc:648] HybridClock initialized: now 1786250985786727 us; error 42 us; skew 500 ppm
I20260809 04:49:45.787051 950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:49:45 dist-test-slave-4tdj krb5kdc[568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250985, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:49:45.789882 950 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:49:45.791257 950 webserver.cc:460] Webserver started at http://127.31.93.2:39199/ using document root <none> and password file <none>
I20260809 04:49:45.791870 950 fs_manager.cc:359] Metadata directory not provided
I20260809 04:49:45.791985 950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:49:45.792292 950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:49:45.795045 950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "30d358559cec414d9904af222bd38360"
format_stamp: "Formatted at 2026-08-09 04:49:45 on dist-test-slave-4tdj"
server_key: "88e3d96339ef99a41c47d89c3bb773b6"
server_key_iv: "44bc83b5133bf5b1652aaf9d4159ac54"
server_key_version: "encryptionkey@0"
I20260809 04:49:45.795789 950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "30d358559cec414d9904af222bd38360"
format_stamp: "Formatted at 2026-08-09 04:49:45 on dist-test-slave-4tdj"
server_key: "88e3d96339ef99a41c47d89c3bb773b6"
server_key_iv: "44bc83b5133bf5b1652aaf9d4159ac54"
server_key_version: "encryptionkey@0"
I20260809 04:49:45.800560 950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 04:49:45.803840 970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:45.805027 950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 04:49:45.805186 950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "30d358559cec414d9904af222bd38360"
format_stamp: "Formatted at 2026-08-09 04:49:45 on dist-test-slave-4tdj"
server_key: "88e3d96339ef99a41c47d89c3bb773b6"
server_key_iv: "44bc83b5133bf5b1652aaf9d4159ac54"
server_key_version: "encryptionkey@0"
I20260809 04:49:45.805366 950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:49:45.819418 950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:49:45.822824 950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:49:45.823069 950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:49:45.823927 950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:49:45.825117 950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:49:45.825203 950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:45.825335 950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:49:45.825392 950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:45.857784 950 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:32949
I20260809 04:49:45.857829 1164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:32949 every 8 connection(s)
I20260809 04:49:45.858989 950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:49:45.859918 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 950
I20260809 04:49:45.860121 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:49:45.860664 32116 external_mini_cluster.cc:1428] Setting key a2c9f34913c5b38e366df2b6119d599c
Aug 09 04:49:45 dist-test-slave-4tdj krb5kdc[568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250985, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
I20260809 04:49:45.879119 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:45.861635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:33145 (local address 127.31.93.62:34565)
0809 04:49:45.861850 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:45.861856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:45.863207 (+ 1351us) server_negotiation.cc:407] Connection header received
0809 04:49:45.865584 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:45.865588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:45.865656 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:45.865734 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:45.867575 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:45.868287 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:45.870028 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:45.870477 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:45.870590 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:45.873808 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:45.873813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:45.873817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:45.873848 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:45.876274 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:45.876935 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:45.876939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:45.876941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:45.877070 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:45.877472 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:45.877476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:45.877478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:45.877723 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:45.877861 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:45.878557 (+ 696us) server_negotiation.cc:299] Negotiation successful
0809 04:49:45.878787 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 04:49:45.879993 1167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:45.862162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:34565 (local address 127.31.93.2:33145)
0809 04:49:45.863030 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:45.863078 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:45.865326 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:45.865962 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:45.865973 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:45.866714 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:45.867430 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:45.867454 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:45.868425 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:45.868430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:45.869888 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:45.869901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:45.870704 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:45.870712 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:45.870962 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:45.871548 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:45.871570 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:45.873611 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:45.876409 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:45.876417 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:45.876432 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:45.876823 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:45.877182 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:45.877187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:45.877191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:45.877368 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:45.877844 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:45.877851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:45.878147 (+ 296us) client_negotiation.cc:769] Sending connection context
0809 04:49:45.878402 (+ 255us) client_negotiation.cc:240] Negotiation successful
0809 04:49:45.878892 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":350,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
I20260809 04:49:45.881366 1165 heartbeater.cc:347] Connected to a master server at 127.31.93.62:34565
I20260809 04:49:45.881661 1165 heartbeater.cc:464] Registering TS with master...
I20260809 04:49:45.882351 1165 heartbeater.cc:511] Master 127.31.93.62:34565 requested a full tablet report, sending...
I20260809 04:49:45.883654 634 ts_manager.cc:194] Registered new tserver with Master: 30d358559cec414d9904af222bd38360 (127.31.93.2:32949)
I20260809 04:49:45.884454 634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:33145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:49:45.900951 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34565
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:49:46.232750 1170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:49:46.233016 1170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:49:46.233119 1170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:49:46.241901 1170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:49:46.242057 1170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:49:46.242226 1170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:49:46.254042 1170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34565
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:49:46.255708 1170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:49:46.257346 1170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:49:46.273288 1178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:46.273347 1180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:46.273428 1182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:49:46.274925 1170 server_base.cc:1034] running on GCE node
I20260809 04:49:46.275678 1170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:49:46.276839 1170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:49:46.278087 1170 hybrid_clock.cc:648] HybridClock initialized: now 1786250986277970 us; error 57 us; skew 500 ppm
I20260809 04:49:46.278491 1170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:49:46 dist-test-slave-4tdj krb5kdc[568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250986, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:49:46.282752 1170 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:49:46.284529 1170 webserver.cc:460] Webserver started at http://127.31.93.3:46309/ using document root <none> and password file <none>
I20260809 04:49:46.285275 1170 fs_manager.cc:359] Metadata directory not provided
I20260809 04:49:46.285400 1170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:49:46.285725 1170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:49:46.288877 1170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "a6a274bff9f44791b7a5c67f7ec097a9"
format_stamp: "Formatted at 2026-08-09 04:49:46 on dist-test-slave-4tdj"
server_key: "ddc0dbc8e748ad060a00ad1aadd7618b"
server_key_iv: "91086cce56007446da0ff07162ecdce0"
server_key_version: "encryptionkey@0"
I20260809 04:49:46.289925 1170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "a6a274bff9f44791b7a5c67f7ec097a9"
format_stamp: "Formatted at 2026-08-09 04:49:46 on dist-test-slave-4tdj"
server_key: "ddc0dbc8e748ad060a00ad1aadd7618b"
server_key_iv: "91086cce56007446da0ff07162ecdce0"
server_key_version: "encryptionkey@0"
I20260809 04:49:46.295670 1170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 04:49:46.299757 1196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:46.301113 1170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 04:49:46.301316 1170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "a6a274bff9f44791b7a5c67f7ec097a9"
format_stamp: "Formatted at 2026-08-09 04:49:46 on dist-test-slave-4tdj"
server_key: "ddc0dbc8e748ad060a00ad1aadd7618b"
server_key_iv: "91086cce56007446da0ff07162ecdce0"
server_key_version: "encryptionkey@0"
I20260809 04:49:46.301595 1170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:49:46.319268 1170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:49:46.323663 1170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:49:46.323947 1170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:49:46.325201 1170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:49:46.326871 1170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:49:46.326958 1170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:46.327097 1170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:49:46.327164 1170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:46.384205 1170 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:36771
I20260809 04:49:46.384310 1392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:36771 every 8 connection(s)
I20260809 04:49:46.385661 1170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:49:46.386891 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 1170
I20260809 04:49:46.387199 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:49:46.387902 32116 external_mini_cluster.cc:1428] Setting key f7eaf1e2cd62872c202a873087fd4ba1
Aug 09 04:49:46 dist-test-slave-4tdj krb5kdc[568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250986, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:46.409415 1401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.388914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:55901 (local address 127.31.93.62:34565)
0809 04:49:46.389392 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:46.389397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:46.391056 (+ 1659us) server_negotiation.cc:407] Connection header received
0809 04:49:46.393610 (+ 2554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:46.393614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:46.393703 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:46.393793 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:46.395643 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.396486 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.398139 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.398849 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.398965 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.403185 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:46.403190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:46.403193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:46.403224 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:46.406549 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:46.407198 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:46.407203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:46.407206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:46.407371 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:46.407712 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:46.407715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:46.407717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:46.408121 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:46.408240 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:46.408815 (+ 575us) server_negotiation.cc:299] Negotiation successful
0809 04:49:46.409022 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":246,"threads_started":1}
I20260809 04:49:46.410248 1403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.389570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:34565 (local address 127.31.93.3:55901)
0809 04:49:46.390902 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:46.390959 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:46.393366 (+ 2407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:46.394022 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:46.394030 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:46.394778 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:46.395513 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:46.395532 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.396605 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.396608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:46.398005 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:46.398017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.399107 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.399116 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:46.399449 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.400026 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:46.400048 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:46.402993 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:46.406661 (+ 3668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:46.406667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:46.406676 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:46.407061 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:46.407454 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:46.407457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:46.407460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:46.407621 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:46.408212 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:46.408217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:46.408512 (+ 295us) client_negotiation.cc:769] Sending connection context
0809 04:49:46.408734 (+ 222us) client_negotiation.cc:240] Negotiation successful
0809 04:49:46.409133 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"thread_start_us":822,"threads_started":1}
I20260809 04:49:46.411580 1393 heartbeater.cc:347] Connected to a master server at 127.31.93.62:34565
I20260809 04:49:46.411845 1393 heartbeater.cc:464] Registering TS with master...
I20260809 04:49:46.412578 1393 heartbeater.cc:511] Master 127.31.93.62:34565 requested a full tablet report, sending...
I20260809 04:49:46.413903 634 ts_manager.cc:194] Registered new tserver with Master: a6a274bff9f44791b7a5c67f7ec097a9 (127.31.93.3:36771)
I20260809 04:49:46.414870 634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:55901
I20260809 04:49:46.415926 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:49:46.429313 935 heartbeater.cc:503] Master 127.31.93.62:34565 was elected leader, sending a full tablet report...
I20260809 04:49:46.440528 1401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.426154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.31.93.62:34565)
0809 04:49:46.426328 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:46.426333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:46.427183 (+ 850us) server_negotiation.cc:407] Connection header received
0809 04:49:46.427362 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:46.427367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:46.427427 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:46.427497 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:46.429932 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.430853 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.432323 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.432743 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.432828 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.435444 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:46.435449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:46.435452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:46.435479 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:46.438182 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:46.438843 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:46.438846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:46.438848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:46.438958 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:46.439307 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:46.439310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:46.439312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:46.439630 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:46.439834 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:46.440120 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 04:49:46.440275 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 04:49:46.450678 634 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:49:46.461464 634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:49:46.507906 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.483092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.31.93.2:32949)
0809 04:49:46.495968 (+ 12876us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:46.495992 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:46.497059 (+ 1067us) server_negotiation.cc:407] Connection header received
0809 04:49:46.497178 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:46.497187 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:46.497453 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:46.497705 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:46.498922 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.501070 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.504561 (+ 3491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.506159 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.506276 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.506483 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:46.507196 (+ 713us) server_negotiation.cc:299] Negotiation successful
0809 04:49:46.507452 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12665,"thread_start_us":12906,"threads_started":1}
I20260809 04:49:46.508092 1418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.483032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:32949 (local address 127.0.0.1:44316)
0809 04:49:46.496725 (+ 13693us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:46.496751 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:46.496900 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:46.497719 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:46.497728 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:46.497764 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:46.498693 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:46.498710 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.501516 (+ 2806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.501521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:46.504324 (+ 2803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:46.504341 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.506616 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.506623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:46.506905 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.506973 (+ 68us) client_negotiation.cc:769] Sending connection context
0809 04:49:46.507091 (+ 118us) client_negotiation.cc:240] Negotiation successful
0809 04:49:46.507350 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13565,"mutex_wait_us":53,"thread_start_us":13620,"threads_started":1}
I20260809 04:49:46.509840 1419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.482655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:33869 (local address 127.0.0.1:40350)
0809 04:49:46.496115 (+ 13460us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:46.496209 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:46.496444 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:46.499641 (+ 3197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:46.499646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:46.499676 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:46.500401 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:46.500418 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.502569 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.502576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:46.506073 (+ 3497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:46.506090 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.508498 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.508507 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:46.508832 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.508878 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 04:49:46.508982 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 04:49:46.509158 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13105,"mutex_wait_us":24,"thread_start_us":13771,"threads_started":1}
I20260809 04:49:46.509948 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.483262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.31.93.1:33869)
0809 04:49:46.498754 (+ 15492us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:46.498763 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:46.498792 (+ 29us) server_negotiation.cc:407] Connection header received
0809 04:49:46.498896 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:46.498905 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:46.499170 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:46.499380 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:46.500694 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.502437 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.506428 (+ 3991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.507916 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.508070 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.509236 (+ 1166us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:46.509331 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 04:49:46.509525 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15287,"thread_start_us":17170,"threads_started":1}
I20260809 04:49:46.513163 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.497611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:36771 (local address 127.0.0.1:42896)
0809 04:49:46.501944 (+ 4333us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:46.501962 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:46.502086 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:46.502767 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:46.502773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:46.502809 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:46.503949 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:46.503970 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.506731 (+ 2761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.506735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:46.509380 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:46.509396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.512246 (+ 2850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.512253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:46.512417 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.512449 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 04:49:46.512548 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 04:49:46.512664 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4174,"mutex_wait_us":129,"thread_start_us":3631,"threads_started":1}
I20260809 04:49:46.512835 1052 tablet_service.cc:1467] Processing CreateTablet for tablet aac50f9a8ef14c1a8988e587b710b9cf (DEFAULT_TABLE table=test-table [id=a063139a5cc5476c97edee64698c122c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:46.514220 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.498118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42896 (local address 127.31.93.3:36771)
0809 04:49:46.500684 (+ 2566us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:46.500693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:46.502207 (+ 1514us) server_negotiation.cc:407] Connection header received
0809 04:49:46.502337 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:46.502346 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:46.502556 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:46.502757 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:46.504278 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.506529 (+ 2251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.509559 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.511701 (+ 2142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.511866 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.512090 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:46.513618 (+ 1528us) server_negotiation.cc:299] Negotiation successful
0809 04:49:46.513776 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2348,"thread_start_us":2129,"threads_started":1}
I20260809 04:49:46.514601 833 tablet_service.cc:1467] Processing CreateTablet for tablet aac50f9a8ef14c1a8988e587b710b9cf (DEFAULT_TABLE table=test-table [id=a063139a5cc5476c97edee64698c122c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:46.515708 1052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aac50f9a8ef14c1a8988e587b710b9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:46.515995 833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aac50f9a8ef14c1a8988e587b710b9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:46.518567 1273 tablet_service.cc:1467] Processing CreateTablet for tablet aac50f9a8ef14c1a8988e587b710b9cf (DEFAULT_TABLE table=test-table [id=a063139a5cc5476c97edee64698c122c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:46.520255 1273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aac50f9a8ef14c1a8988e587b710b9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:46.529618 1424 tablet_bootstrap.cc:492] T aac50f9a8ef14c1a8988e587b710b9cf P 5a9abd86d80b41f49d3fd663906ab1f4: Bootstrap starting.
I20260809 04:49:46.533882 1424 tablet_bootstrap.cc:654] T aac50f9a8ef14c1a8988e587b710b9cf P 5a9abd86d80b41f49d3fd663906ab1f4: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:46.534045 1426 tablet_bootstrap.cc:492] T aac50f9a8ef14c1a8988e587b710b9cf P 30d358559cec414d9904af222bd38360: Bootstrap starting.
I20260809 04:49:46.535714 1424 log.cc:824] T aac50f9a8ef14c1a8988e587b710b9cf P 5a9abd86d80b41f49d3fd663906ab1f4: Log is configured to *not* fsync() on all Append() calls
I20260809 04:49:46.537082 1427 tablet_bootstrap.cc:492] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9: Bootstrap starting.
I20260809 04:49:46.538761 1426 tablet_bootstrap.cc:654] T aac50f9a8ef14c1a8988e587b710b9cf P 30d358559cec414d9904af222bd38360: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:46.540807 1424 tablet_bootstrap.cc:492] T aac50f9a8ef14c1a8988e587b710b9cf P 5a9abd86d80b41f49d3fd663906ab1f4: No bootstrap required, opened a new log
I20260809 04:49:46.540904 1426 log.cc:824] T aac50f9a8ef14c1a8988e587b710b9cf P 30d358559cec414d9904af222bd38360: Log is configured to *not* fsync() on all Append() calls
I20260809 04:49:46.541097 1424 ts_tablet_manager.cc:1397] T aac50f9a8ef14c1a8988e587b710b9cf P 5a9abd86d80b41f49d3fd663906ab1f4: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260809 04:49:46.541728 1427 tablet_bootstrap.cc:654] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:46.543514 1427 log.cc:824] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9: Log is configured to *not* fsync() on all Append() calls
I20260809 04:49:46.544502 1426 tablet_bootstrap.cc:492] T aac50f9a8ef14c1a8988e587b710b9cf P 30d358559cec414d9904af222bd38360: No bootstrap required, opened a new log
I20260809 04:49:46.544751 1426 ts_tablet_manager.cc:1397] T aac50f9a8ef14c1a8988e587b710b9cf P 30d358559cec414d9904af222bd38360: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260809 04:49:46.546430 1427 tablet_bootstrap.cc:492] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9: No bootstrap required, opened a new log
I20260809 04:49:46.546801 1427 ts_tablet_manager.cc:1397] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260809 04:49:46.553619 1424 raft_consensus.cc:348] T aac50f9a8ef14c1a8988e587b710b9cf P 5a9abd86d80b41f49d3fd663906ab1f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9abd86d80b41f49d3fd663906ab1f4" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33869 } } peers { permanent_uuid: "30d358559cec414d9904af222bd38360" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 32949 } } peers { permanent_uuid: "a6a274bff9f44791b7a5c67f7ec097a9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36771 } }
I20260809 04:49:46.554061 1424 raft_consensus.cc:374] T aac50f9a8ef14c1a8988e587b710b9cf P 5a9abd86d80b41f49d3fd663906ab1f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:46.554208 1424 raft_consensus.cc:729] T aac50f9a8ef14c1a8988e587b710b9cf P 5a9abd86d80b41f49d3fd663906ab1f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9abd86d80b41f49d3fd663906ab1f4, State: Initialized, Role: FOLLOWER
I20260809 04:49:46.554952 1424 consensus_queue.cc:260] T aac50f9a8ef14c1a8988e587b710b9cf P 5a9abd86d80b41f49d3fd663906ab1f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9abd86d80b41f49d3fd663906ab1f4" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33869 } } peers { permanent_uuid: "30d358559cec414d9904af222bd38360" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 32949 } } peers { permanent_uuid: "a6a274bff9f44791b7a5c67f7ec097a9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36771 } }
I20260809 04:49:46.556893 1424 ts_tablet_manager.cc:1428] T aac50f9a8ef14c1a8988e587b710b9cf P 5a9abd86d80b41f49d3fd663906ab1f4: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260809 04:49:46.556818 1427 raft_consensus.cc:348] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9abd86d80b41f49d3fd663906ab1f4" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33869 } } peers { permanent_uuid: "30d358559cec414d9904af222bd38360" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 32949 } } peers { permanent_uuid: "a6a274bff9f44791b7a5c67f7ec097a9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36771 } }
I20260809 04:49:46.557415 1427 raft_consensus.cc:374] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:46.557597 1427 raft_consensus.cc:729] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a274bff9f44791b7a5c67f7ec097a9, State: Initialized, Role: FOLLOWER
I20260809 04:49:46.557965 1426 raft_consensus.cc:348] T aac50f9a8ef14c1a8988e587b710b9cf P 30d358559cec414d9904af222bd38360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9abd86d80b41f49d3fd663906ab1f4" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33869 } } peers { permanent_uuid: "30d358559cec414d9904af222bd38360" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 32949 } } peers { permanent_uuid: "a6a274bff9f44791b7a5c67f7ec097a9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36771 } }
I20260809 04:49:46.558460 1426 raft_consensus.cc:374] T aac50f9a8ef14c1a8988e587b710b9cf P 30d358559cec414d9904af222bd38360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:46.558233 1427 consensus_queue.cc:260] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9abd86d80b41f49d3fd663906ab1f4" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33869 } } peers { permanent_uuid: "30d358559cec414d9904af222bd38360" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 32949 } } peers { permanent_uuid: "a6a274bff9f44791b7a5c67f7ec097a9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36771 } }
I20260809 04:49:46.558614 1426 raft_consensus.cc:729] T aac50f9a8ef14c1a8988e587b710b9cf P 30d358559cec414d9904af222bd38360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d358559cec414d9904af222bd38360, State: Initialized, Role: FOLLOWER
I20260809 04:49:46.559325 1426 consensus_queue.cc:260] T aac50f9a8ef14c1a8988e587b710b9cf P 30d358559cec414d9904af222bd38360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9abd86d80b41f49d3fd663906ab1f4" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33869 } } peers { permanent_uuid: "30d358559cec414d9904af222bd38360" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 32949 } } peers { permanent_uuid: "a6a274bff9f44791b7a5c67f7ec097a9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36771 } }
I20260809 04:49:46.559860 1393 heartbeater.cc:503] Master 127.31.93.62:34565 was elected leader, sending a full tablet report...
I20260809 04:49:46.560711 1427 ts_tablet_manager.cc:1428] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260809 04:49:46.561890 1165 heartbeater.cc:503] Master 127.31.93.62:34565 was elected leader, sending a full tablet report...
I20260809 04:49:46.563895 1426 ts_tablet_manager.cc:1428] T aac50f9a8ef14c1a8988e587b710b9cf P 30d358559cec414d9904af222bd38360: Time spent starting tablet: real 0.019s user 0.008s sys 0.011s
W20260809 04:49:46.610868 1166 tablet.cc:2369] T aac50f9a8ef14c1a8988e587b710b9cf P 30d358559cec414d9904af222bd38360: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:49:46.638226 1394 tablet.cc:2369] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:49:46.651330 941 tablet.cc:2369] T aac50f9a8ef14c1a8988e587b710b9cf P 5a9abd86d80b41f49d3fd663906ab1f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:49:46.655736 1433 raft_consensus.cc:482] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:49:46.656116 1433 raft_consensus.cc:504] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9abd86d80b41f49d3fd663906ab1f4" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33869 } } peers { permanent_uuid: "30d358559cec414d9904af222bd38360" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 32949 } } peers { permanent_uuid: "a6a274bff9f44791b7a5c67f7ec097a9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36771 } }
I20260809 04:49:46.658457 1433 leader_election.cc:288] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a9abd86d80b41f49d3fd663906ab1f4 (127.31.93.1:33869), 30d358559cec414d9904af222bd38360 (127.31.93.2:32949)
I20260809 04:49:46.666872 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.659115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:38785 (local address 127.31.93.2:32949)
0809 04:49:46.659468 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:46.659474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:46.659493 (+ 19us) server_negotiation.cc:407] Connection header received
0809 04:49:46.659661 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:46.659665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:46.659727 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:46.659800 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:46.660359 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.661506 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.664332 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.665820 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.665958 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.666055 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:46.666489 (+ 434us) server_negotiation.cc:299] Negotiation successful
0809 04:49:46.666620 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260809 04:49:46.666958 1403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.658862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:32949 (local address 127.31.93.3:38785)
0809 04:49:46.659383 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:46.659404 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:46.659526 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:46.659835 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:46.659839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:46.659872 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:46.660220 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:46.660230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.661616 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.661619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:46.664185 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:46.664200 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.665982 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.665986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:46.666273 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.666338 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 04:49:46.666433 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 04:49:46.666572 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331}
I20260809 04:49:46.667182 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.659115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:48359 (local address 127.31.93.1:33869)
0809 04:49:46.659374 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:46.659380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:46.660405 (+ 1025us) server_negotiation.cc:407] Connection header received
0809 04:49:46.660527 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:46.660533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:46.660613 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:46.660708 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:46.661429 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.662241 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.664760 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.666139 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.666264 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.666386 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:46.666820 (+ 434us) server_negotiation.cc:299] Negotiation successful
0809 04:49:46.666956 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 04:49:46.667732 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.658873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:33869 (local address 127.31.93.3:48359)
0809 04:49:46.660308 (+ 1435us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:46.660323 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:46.660425 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:46.660709 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:46.660711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:46.660727 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:46.661243 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:46.661253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.662416 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.662422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:46.664614 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:46.664625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.666394 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.666401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:46.666628 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.666670 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 04:49:46.666769 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 04:49:46.666889 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1331,"mutex_wait_us":31,"thread_start_us":874,"threads_started":1}
I20260809 04:49:46.668277 1085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac50f9a8ef14c1a8988e587b710b9cf" candidate_uuid: "a6a274bff9f44791b7a5c67f7ec097a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30d358559cec414d9904af222bd38360" is_pre_election: true
I20260809 04:49:46.668776 856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac50f9a8ef14c1a8988e587b710b9cf" candidate_uuid: "a6a274bff9f44791b7a5c67f7ec097a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9abd86d80b41f49d3fd663906ab1f4" is_pre_election: true
I20260809 04:49:46.669139 1085 raft_consensus.cc:2454] T aac50f9a8ef14c1a8988e587b710b9cf P 30d358559cec414d9904af222bd38360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a274bff9f44791b7a5c67f7ec097a9 in term 0.
I20260809 04:49:46.669333 856 raft_consensus.cc:2454] T aac50f9a8ef14c1a8988e587b710b9cf P 5a9abd86d80b41f49d3fd663906ab1f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a274bff9f44791b7a5c67f7ec097a9 in term 0.
I20260809 04:49:46.670177 1201 leader_election.cc:302] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a9abd86d80b41f49d3fd663906ab1f4, a6a274bff9f44791b7a5c67f7ec097a9; no voters:
I20260809 04:49:46.670771 1433 raft_consensus.cc:2781] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:49:46.670997 1433 raft_consensus.cc:482] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:49:46.671190 1433 raft_consensus.cc:3037] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:46.674208 1433 raft_consensus.cc:504] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9abd86d80b41f49d3fd663906ab1f4" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33869 } } peers { permanent_uuid: "30d358559cec414d9904af222bd38360" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 32949 } } peers { permanent_uuid: "a6a274bff9f44791b7a5c67f7ec097a9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36771 } }
I20260809 04:49:46.675045 1433 leader_election.cc:288] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [CANDIDATE]: Term 1 election: Requested vote from peers 5a9abd86d80b41f49d3fd663906ab1f4 (127.31.93.1:33869), 30d358559cec414d9904af222bd38360 (127.31.93.2:32949)
I20260809 04:49:46.675328 856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac50f9a8ef14c1a8988e587b710b9cf" candidate_uuid: "a6a274bff9f44791b7a5c67f7ec097a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9abd86d80b41f49d3fd663906ab1f4"
I20260809 04:49:46.675446 1085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aac50f9a8ef14c1a8988e587b710b9cf" candidate_uuid: "a6a274bff9f44791b7a5c67f7ec097a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30d358559cec414d9904af222bd38360"
I20260809 04:49:46.675500 856 raft_consensus.cc:3037] T aac50f9a8ef14c1a8988e587b710b9cf P 5a9abd86d80b41f49d3fd663906ab1f4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:46.675632 1085 raft_consensus.cc:3037] T aac50f9a8ef14c1a8988e587b710b9cf P 30d358559cec414d9904af222bd38360 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:46.678090 856 raft_consensus.cc:2454] T aac50f9a8ef14c1a8988e587b710b9cf P 5a9abd86d80b41f49d3fd663906ab1f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a274bff9f44791b7a5c67f7ec097a9 in term 1.
I20260809 04:49:46.678565 1201 leader_election.cc:302] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a9abd86d80b41f49d3fd663906ab1f4, a6a274bff9f44791b7a5c67f7ec097a9; no voters:
I20260809 04:49:46.678911 1433 raft_consensus.cc:2781] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:49:46.679042 1085 raft_consensus.cc:2454] T aac50f9a8ef14c1a8988e587b710b9cf P 30d358559cec414d9904af222bd38360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a274bff9f44791b7a5c67f7ec097a9 in term 1.
I20260809 04:49:46.682582 1433 raft_consensus.cc:686] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [term 1 LEADER]: Becoming Leader. State: Replica: a6a274bff9f44791b7a5c67f7ec097a9, State: Running, Role: LEADER
I20260809 04:49:46.683234 1433 consensus_queue.cc:237] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9abd86d80b41f49d3fd663906ab1f4" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33869 } } peers { permanent_uuid: "30d358559cec414d9904af222bd38360" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 32949 } } peers { permanent_uuid: "a6a274bff9f44791b7a5c67f7ec097a9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36771 } }
I20260809 04:49:46.688472 631 catalog_manager.cc:5495] T aac50f9a8ef14c1a8988e587b710b9cf P a6a274bff9f44791b7a5c67f7ec097a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6a274bff9f44791b7a5c67f7ec097a9 (127.31.93.3). New cstate: current_term: 1 leader_uuid: "a6a274bff9f44791b7a5c67f7ec097a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9abd86d80b41f49d3fd663906ab1f4" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30d358559cec414d9904af222bd38360" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 32949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a274bff9f44791b7a5c67f7ec097a9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36771 } health_report { overall_health: HEALTHY } } }
Aug 09 04:49:46 dist-test-slave-4tdj krb5kdc[568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:49:46.756477 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.738376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.31.93.1:33869)
0809 04:49:46.738600 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:46.738606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:46.739553 (+ 947us) server_negotiation.cc:407] Connection header received
0809 04:49:46.739723 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:46.739727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:46.739792 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:46.739908 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:46.741641 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.742591 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.744017 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.744498 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.744602 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.750053 (+ 5451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:46.750075 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:46.750108 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:46.750148 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:46.753672 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:46.754477 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:46.754488 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:46.754492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:46.754626 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:46.754987 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:46.754993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:46.754996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:46.755344 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:46.755624 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:46.755938 (+ 314us) server_negotiation.cc:299] Negotiation successful
0809 04:49:46.756152 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Aug 09 04:49:46 dist-test-slave-4tdj krb5kdc[568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250986, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 04:49:46 dist-test-slave-4tdj krb5kdc[568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250986, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:49:46.806140 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.791393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.31.93.1:33869)
0809 04:49:46.791626 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:46.791633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:46.792000 (+ 367us) server_negotiation.cc:407] Connection header received
0809 04:49:46.792171 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:46.792175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:46.792242 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:46.792329 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:46.794415 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.795116 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.796531 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.796910 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.796999 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.801017 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:46.801022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:46.801025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:46.801057 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:46.803661 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:46.804210 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:46.804213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:46.804216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:46.804425 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:46.804877 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:46.804881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:46.804883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:46.805162 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:46.805457 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:46.805569 (+ 112us) server_negotiation.cc:299] Negotiation successful
0809 04:49:46.805780 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Aug 09 04:49:46 dist-test-slave-4tdj krb5kdc[568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 04:49:46 dist-test-slave-4tdj krb5kdc[568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:49:46.855165 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.838975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.31.93.1:33869)
0809 04:49:46.839199 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:46.839205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:46.839507 (+ 302us) server_negotiation.cc:407] Connection header received
0809 04:49:46.839701 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:46.839706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:46.839768 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:46.839847 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:46.841779 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.842502 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.844304 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.844716 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.844802 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.849588 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:46.849593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:46.849597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:46.849630 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:46.852718 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:46.853252 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:46.853256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:46.853258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:46.853366 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:46.853656 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:46.853660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:46.853662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:46.854025 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:46.854383 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:46.854511 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 04:49:46.854692 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260809 04:49:46.857405 799 tablet_service.cc:2908] Rejecting scan request for tablet aac50f9a8ef14c1a8988e587b710b9cf: Uninitialized: clean time has not yet been initialized
I20260809 04:49:46.862174 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 735
I20260809 04:49:46.876391 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.863287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.31.93.1:33869)
0809 04:49:46.863457 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:46.863462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:46.863823 (+ 361us) server_negotiation.cc:407] Connection header received
0809 04:49:46.863925 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:46.863930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:46.864007 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:46.864103 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:46.865170 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.866186 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.867527 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.868233 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.868416 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.870538 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:46.870545 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:46.870550 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:46.870584 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:46.874141 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:46.874865 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:46.874869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:46.874871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:46.874975 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:46.875243 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:46.875245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:46.875247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:46.875563 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:46.875797 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:46.875940 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 04:49:46.876103 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 04:49:46.876957 927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:49:46.964856 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 735
I20260809 04:49:46.988811 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 950
Aug 09 04:49:46 dist-test-slave-4tdj krb5kdc[568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:49:47.002954 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:46.989568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44324 (local address 127.31.93.2:32949)
0809 04:49:46.989749 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:46.989754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:46.989773 (+ 19us) server_negotiation.cc:407] Connection header received
0809 04:49:46.989868 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:46.989872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:46.989935 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:46.990013 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:46.991174 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.991994 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.993566 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:46.994007 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:46.994110 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:46.997916 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:46.997929 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:46.997945 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:46.997979 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:47.000700 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:47.001286 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:47.001293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:47.001295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:47.001416 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:47.001691 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:47.001693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:47.001695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:47.002036 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:47.002279 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:47.002474 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 04:49:47.002635 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:49:47.003609 1157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:49:47.096699 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 950
I20260809 04:49:47.118623 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 1170
Aug 09 04:49:47 dist-test-slave-4tdj krb5kdc[568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:49:47.132472 1470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:47.119387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.31.93.3:36771)
0809 04:49:47.119831 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:47.119835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:47.119853 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:49:47.119908 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:47.119912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:47.119982 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:47.120067 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:47.121213 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:47.121873 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:47.123185 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:47.123587 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:47.123678 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:47.127386 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:47.127400 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:47.127417 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:47.127449 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:47.130066 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:47.130716 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:47.130723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:47.130726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:47.130864 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:47.131209 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:47.131213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:47.131215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:47.131593 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:47.131811 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:47.132023 (+ 212us) server_negotiation.cc:299] Negotiation successful
0809 04:49:47.132192 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":216,"threads_started":1}
I20260809 04:49:47.133111 1387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:49:47.219883 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 1170
I20260809 04:49:47.240609 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 585
I20260809 04:49:47.241703 692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:49:47.346791 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 585
2026-08-09T04:49:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4265 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:49:47 dist-test-slave-4tdj krb5kdc[1483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:49:47 dist-test-slave-4tdj krb5kdc[1483](info): set up 2 sockets
Aug 09 04:49:47 dist-test-slave-4tdj krb5kdc[1483](info): commencing operation
krb5kdc: starting...
W20260809 04:49:48.527745 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:49:48 dist-test-slave-4tdj krb5kdc[1483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:49:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:49:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:49:48.695906 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:44229
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:44229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:49:49.005460 1502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:49:49.005786 1502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:49:49.005918 1502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:49:49.013113 1502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:49:49.013221 1502 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 04:49:49.013290 1502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:49:49.013375 1502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:49:49.013439 1502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:49:49.013528 1502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:49:49.023712 1502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:44229
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:44229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:49:49.025327 1502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:49:49.026921 1502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:49:49.038182 1523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:49.038153 1517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:49.038560 1515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:49:49.038960 1502 server_base.cc:1034] running on GCE node
I20260809 04:49:49.039739 1502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:49:49.041041 1502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:49:49.042248 1502 hybrid_clock.cc:648] HybridClock initialized: now 1786250989042176 us; error 52 us; skew 500 ppm
I20260809 04:49:49.042553 1502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:49:49 dist-test-slave-4tdj krb5kdc[1483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250989, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:49:49.045444 1502 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:49:49.046800 1502 webserver.cc:460] Webserver started at http://127.31.93.62:37221/ using document root <none> and password file <none>
I20260809 04:49:49.047429 1502 fs_manager.cc:359] Metadata directory not provided
I20260809 04:49:49.047514 1502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:49:49.047788 1502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:49:49.050931 1502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "3a6bea19994647a583bc844a43a18af9"
format_stamp: "Formatted at 2026-08-09 04:49:49 on dist-test-slave-4tdj"
server_key: "0048db3dd6cb27f39e5a8f73060c967b"
server_key_iv: "0fdb3ee3fabbd808ca4dc8aac7e28339"
server_key_version: "encryptionkey@0"
I20260809 04:49:49.051623 1502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "3a6bea19994647a583bc844a43a18af9"
format_stamp: "Formatted at 2026-08-09 04:49:49 on dist-test-slave-4tdj"
server_key: "0048db3dd6cb27f39e5a8f73060c967b"
server_key_iv: "0fdb3ee3fabbd808ca4dc8aac7e28339"
server_key_version: "encryptionkey@0"
I20260809 04:49:49.056337 1502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:49:49.059489 1534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:49.060673 1502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:49:49.060804 1502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "3a6bea19994647a583bc844a43a18af9"
format_stamp: "Formatted at 2026-08-09 04:49:49 on dist-test-slave-4tdj"
server_key: "0048db3dd6cb27f39e5a8f73060c967b"
server_key_iv: "0fdb3ee3fabbd808ca4dc8aac7e28339"
server_key_version: "encryptionkey@0"
I20260809 04:49:49.060966 1502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:49:49.075754 1502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:49:49.079190 1502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:49:49.079386 1502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:49:49.098554 1502 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:44229
I20260809 04:49:49.098577 1636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:44229 every 8 connection(s)
I20260809 04:49:49.099762 1502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:49:49.103183 1637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:49.109220 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 1502
I20260809 04:49:49.109418 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:49:49.110044 32116 external_mini_cluster.cc:1428] Setting key 2a62f117fce10dd9b470a5592c26bc51
I20260809 04:49:49.113215 1637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9: Bootstrap starting.
I20260809 04:49:49.116681 1637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:49.117712 1637 log.cc:824] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9: Log is configured to *not* fsync() on all Append() calls
I20260809 04:49:49.120489 1637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9: No bootstrap required, opened a new log
Aug 09 04:49:49 dist-test-slave-4tdj krb5kdc[1483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:49.128607 1637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6bea19994647a583bc844a43a18af9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44229 } }
I20260809 04:49:49.128952 1637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:49.129055 1637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a6bea19994647a583bc844a43a18af9, State: Initialized, Role: FOLLOWER
I20260809 04:49:49.129598 1637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6bea19994647a583bc844a43a18af9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44229 } }
I20260809 04:49:49.129825 1637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:49:49.129985 1637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:49:49.130163 1637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:49.131670 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:49.111861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.31.93.62:44229)
0809 04:49:49.112597 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:49.112607 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:49.112645 (+ 38us) server_negotiation.cc:407] Connection header received
0809 04:49:49.114269 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:49.114317 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:49.114680 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:49.114981 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:49.116703 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:49.117994 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:49.119508 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:49.120134 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:49.120282 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:49.124825 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:49.124835 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:49.124848 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:49.124890 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:49.127776 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:49.128415 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:49.128421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:49.128426 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:49.128578 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:49.128914 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:49.128918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:49.128921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:49.129571 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:49.129859 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:49.130249 (+ 390us) server_negotiation.cc:299] Negotiation successful
0809 04:49:49.130667 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":333,"threads_started":1}
I20260809 04:49:49.132516 1637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6bea19994647a583bc844a43a18af9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44229 } }
I20260809 04:49:49.133003 1637 leader_election.cc:302] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a6bea19994647a583bc844a43a18af9; no voters:
I20260809 04:49:49.133620 1637 leader_election.cc:288] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:49:49.133740 1647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:49:49.134810 1647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a6bea19994647a583bc844a43a18af9, State: Running, Role: LEADER
I20260809 04:49:49.135275 1647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6bea19994647a583bc844a43a18af9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44229 } }
I20260809 04:49:49.135877 1637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:49:49.137534 1649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a6bea19994647a583bc844a43a18af9. Latest consensus state: current_term: 1 leader_uuid: "3a6bea19994647a583bc844a43a18af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6bea19994647a583bc844a43a18af9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44229 } } }
I20260809 04:49:49.137919 1649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [sys.catalog]: This master's current role is: LEADER
I20260809 04:49:49.138752 1652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:49:49.138577 1648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a6bea19994647a583bc844a43a18af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6bea19994647a583bc844a43a18af9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44229 } } }
I20260809 04:49:49.138965 1648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9 [sys.catalog]: This master's current role is: LEADER
I20260809 04:49:49.143584 1652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:49:49.151538 1652 catalog_manager.cc:1347] Generated new cluster ID: 3aacc5cf54c24fd18bf8f7e37349d17b
I20260809 04:49:49.151628 1652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:49:49.171569 1652 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:49:49.172685 1652 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:49:49.178372 1652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9: Generated new TSK 0
I20260809 04:49:49.178989 1652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:49:49.219602 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44229
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:49:49.531585 1684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:49:49.531841 1684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:49:49.531968 1684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:49:49.538951 1684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:49:49.539047 1684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:49:49.539151 1684 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:49:49.539256 1684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:49:49.548691 1684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.93.62:44229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:49:49.549923 1684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:49:49.551275 1684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:49:49.563249 1690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:49.563284 1693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:49.563320 1691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:49:49.564337 1684 server_base.cc:1034] running on GCE node
I20260809 04:49:49.564921 1684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:49:49.565785 1684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:49:49.566987 1684 hybrid_clock.cc:648] HybridClock initialized: now 1786250989566920 us; error 37 us; skew 500 ppm
I20260809 04:49:49.567267 1684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:49:49 dist-test-slave-4tdj krb5kdc[1483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250989, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:49:49.570194 1684 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:49:49.571485 1684 webserver.cc:460] Webserver started at http://127.31.93.1:46489/ using document root <none> and password file <none>
I20260809 04:49:49.572141 1684 fs_manager.cc:359] Metadata directory not provided
I20260809 04:49:49.572239 1684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:49:49.572528 1684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:49:49.575342 1684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "cb131a95dd3244b5a1ea0cd7aba4e105"
format_stamp: "Formatted at 2026-08-09 04:49:49 on dist-test-slave-4tdj"
server_key: "a20985a582800c5dedb265e938919595"
server_key_iv: "c777cf7fffc4598d40cb7a5269ba14a4"
server_key_version: "encryptionkey@0"
I20260809 04:49:49.576122 1684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "cb131a95dd3244b5a1ea0cd7aba4e105"
format_stamp: "Formatted at 2026-08-09 04:49:49 on dist-test-slave-4tdj"
server_key: "a20985a582800c5dedb265e938919595"
server_key_iv: "c777cf7fffc4598d40cb7a5269ba14a4"
server_key_version: "encryptionkey@0"
I20260809 04:49:49.581053 1684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 04:49:49.584285 1703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:49.585495 1684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:49:49.585649 1684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "cb131a95dd3244b5a1ea0cd7aba4e105"
format_stamp: "Formatted at 2026-08-09 04:49:49 on dist-test-slave-4tdj"
server_key: "a20985a582800c5dedb265e938919595"
server_key_iv: "c777cf7fffc4598d40cb7a5269ba14a4"
server_key_version: "encryptionkey@0"
I20260809 04:49:49.585837 1684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:49:49.601851 1684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:49:49.605475 1684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:49:49.605713 1684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:49:49.608341 1684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:49:49.608443 1684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:49.608582 1684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:49:49.608654 1684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:49:49 dist-test-slave-4tdj krb5kdc[1483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250989, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:49.630863 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:49.609046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:36969 (local address 127.31.93.62:44229)
0809 04:49:49.609209 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:49.609213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:49.610538 (+ 1325us) server_negotiation.cc:407] Connection header received
0809 04:49:49.612570 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:49.612576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:49.612675 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:49.612798 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:49.614765 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:49.615874 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:49.618784 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:49.619439 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:49.619578 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:49.623864 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:49.623869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:49.623873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:49.623906 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:49.627128 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:49.628006 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:49.628010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:49.628012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:49.628150 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:49.628782 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:49.628788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:49.628791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:49.629171 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:49.629328 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:49.630180 (+ 852us) server_negotiation.cc:299] Negotiation successful
0809 04:49:49.630466 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 04:49:49.631884 1721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:49.609582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44229 (local address 127.31.93.1:36969)
0809 04:49:49.610390 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:49.610435 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:49.612231 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:49.613046 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:49.613057 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:49.613774 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:49.614558 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:49.614581 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:49.616090 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:49.616095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:49.618566 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:49.618582 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:49.619706 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:49.619714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:49.619999 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:49.620646 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:49.620689 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:49.623644 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:49.627288 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:49.627301 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:49.627316 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:49.627889 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:49.628278 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:49.628283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:49.628287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:49.628600 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:49.629335 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:49.629343 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:49.629741 (+ 398us) client_negotiation.cc:769] Sending connection context
0809 04:49:49.630055 (+ 314us) client_negotiation.cc:240] Negotiation successful
0809 04:49:49.630552 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":343,"threads_started":1}
I20260809 04:49:49.651482 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:49.638749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:44233 (local address 127.31.93.62:44229)
0809 04:49:49.638928 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:49.638934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:49.639529 (+ 595us) server_negotiation.cc:407] Connection header received
0809 04:49:49.639703 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:49.639707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:49.639764 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:49.639847 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:49.641024 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:49.641819 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:49.643609 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:49.644001 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:49.644104 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:49.645782 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:49.645786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:49.645789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:49.645818 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:49.649031 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:49.649635 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:49.649640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:49.649644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:49.649768 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:49.650086 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:49.650091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:49.650095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:49.650414 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:49.650554 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:49.650940 (+ 386us) server_negotiation.cc:299] Negotiation successful
0809 04:49:49.651133 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:49:49.651557 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:49.638753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44229 (local address 127.31.93.1:44233)
0809 04:49:49.639406 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:49.639428 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:49.639582 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:49.639899 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:49.639904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:49.640457 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:49.640884 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:49.640897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:49.641947 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:49.641956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:49.643484 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:49.643499 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:49.644168 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:49.644175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:49.644338 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:49.644933 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:49.644949 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:49.645626 (+ 677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:49.649151 (+ 3525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:49.649157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:49.649161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:49.649528 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:49.649869 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:49.649873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:49.649878 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:49.649992 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:49.650525 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:49.650529 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:49.650715 (+ 186us) client_negotiation.cc:769] Sending connection context
0809 04:49:49.650838 (+ 123us) client_negotiation.cc:240] Negotiation successful
0809 04:49:49.651151 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":335,"threads_started":1}
I20260809 04:49:49.659233 1684 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:34927
I20260809 04:49:49.659260 1895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:34927 every 8 connection(s)
I20260809 04:49:49.660352 1684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:49:49.662158 1898 heartbeater.cc:347] Connected to a master server at 127.31.93.62:44229
I20260809 04:49:49.662420 1898 heartbeater.cc:464] Registering TS with master...
I20260809 04:49:49.662989 1898 heartbeater.cc:511] Master 127.31.93.62:44229 requested a full tablet report, sending...
I20260809 04:49:49.664871 1560 ts_manager.cc:194] Registered new tserver with Master: cb131a95dd3244b5a1ea0cd7aba4e105 (127.31.93.1:34927)
I20260809 04:49:49.665145 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 1684
I20260809 04:49:49.665390 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:49:49.666172 32116 external_mini_cluster.cc:1428] Setting key 8823af8fa8aa2677c7984fc312bbbfbf
I20260809 04:49:49.666865 1560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:36969
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:49:49.704497 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44229
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:49:50.015144 1903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:49:50.015431 1903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:49:50.015574 1903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:49:50.023170 1903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:49:50.023280 1903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:49:50.023406 1903 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:49:50.023517 1903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:49:50.036976 1903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.93.62:44229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:49:50.038671 1903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:49:50.040547 1903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:49:50.056304 1912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:50.056392 1913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:50.057427 1917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:49:50.057823 1903 server_base.cc:1034] running on GCE node
I20260809 04:49:50.058606 1903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:49:50.059700 1903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:49:50.060899 1903 hybrid_clock.cc:648] HybridClock initialized: now 1786250990060822 us; error 54 us; skew 500 ppm
I20260809 04:49:50.061179 1903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:49:50 dist-test-slave-4tdj krb5kdc[1483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:49:50.064231 1903 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:49:50.065656 1903 webserver.cc:460] Webserver started at http://127.31.93.2:43235/ using document root <none> and password file <none>
I20260809 04:49:50.066352 1903 fs_manager.cc:359] Metadata directory not provided
I20260809 04:49:50.066447 1903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:49:50.066721 1903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:49:50.069492 1903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "8f068481bea1413db2de885a33564966"
format_stamp: "Formatted at 2026-08-09 04:49:50 on dist-test-slave-4tdj"
server_key: "e77dbf2971fd4b90414592c6992b752c"
server_key_iv: "1a3f4fb96540acabfa1b6ed0cca3b421"
server_key_version: "encryptionkey@0"
I20260809 04:49:50.070235 1903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "8f068481bea1413db2de885a33564966"
format_stamp: "Formatted at 2026-08-09 04:49:50 on dist-test-slave-4tdj"
server_key: "e77dbf2971fd4b90414592c6992b752c"
server_key_iv: "1a3f4fb96540acabfa1b6ed0cca3b421"
server_key_version: "encryptionkey@0"
I20260809 04:49:50.075126 1903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 04:49:50.078272 1929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:50.079506 1903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260809 04:49:50.079633 1903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "8f068481bea1413db2de885a33564966"
format_stamp: "Formatted at 2026-08-09 04:49:50 on dist-test-slave-4tdj"
server_key: "e77dbf2971fd4b90414592c6992b752c"
server_key_iv: "1a3f4fb96540acabfa1b6ed0cca3b421"
server_key_version: "encryptionkey@0"
I20260809 04:49:50.079795 1903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:49:50.098101 1903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:49:50.102053 1903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:49:50.102339 1903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:49:50.104950 1903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:49:50.105041 1903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:50.105185 1903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:49:50.105257 1903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:49:50 dist-test-slave-4tdj krb5kdc[1483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:50.128974 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:50.106067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:42579 (local address 127.31.93.62:44229)
0809 04:49:50.106251 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:50.106257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:50.107362 (+ 1105us) server_negotiation.cc:407] Connection header received
0809 04:49:50.110219 (+ 2857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:50.110228 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:50.110345 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:50.110423 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:50.112732 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.113942 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.116406 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.117083 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.117224 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:50.121949 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:50.121956 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:50.121960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:50.122002 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:50.125158 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:50.126069 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:50.126075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:50.126087 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:50.126237 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:50.126874 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:50.126879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:50.126882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:50.127219 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:50.127350 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:50.128363 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0809 04:49:50.128645 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 04:49:50.130326 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:50.106228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44229 (local address 127.31.93.2:42579)
0809 04:49:50.107190 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:50.107237 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:50.109860 (+ 2623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:50.110975 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:50.110987 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:50.111719 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:50.112532 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:50.112557 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.114151 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.114156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:50.116240 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:50.116257 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.117338 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.117346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:50.117636 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:50.118343 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:50.118377 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:50.121642 (+ 3265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:50.125312 (+ 3670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:50.125321 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:50.125337 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:50.125942 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:50.126407 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:50.126412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:50.126416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:50.126734 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:50.127448 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:50.127458 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:50.127891 (+ 433us) client_negotiation.cc:769] Sending connection context
0809 04:49:50.128235 (+ 344us) client_negotiation.cc:240] Negotiation successful
0809 04:49:50.128771 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":389,"threads_started":1}
I20260809 04:49:50.152242 1903 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:35339
I20260809 04:49:50.152310 2124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:35339 every 8 connection(s)
I20260809 04:49:50.153632 1903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:49:50.155756 2126 heartbeater.cc:347] Connected to a master server at 127.31.93.62:44229
I20260809 04:49:50.156160 2126 heartbeater.cc:464] Registering TS with master...
I20260809 04:49:50.156361 2091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:50.141866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44229 (local address 127.31.93.2:40139)
0809 04:49:50.142773 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:50.142792 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:50.142902 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:50.143255 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:50.143260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:50.143658 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:50.143938 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:50.143947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.145344 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.145354 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:50.146506 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:50.146548 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.147491 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.147500 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:50.147720 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:50.148380 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:50.148407 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:50.149717 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:50.153220 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:50.153225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:50.153229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:50.153740 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:50.154066 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:50.154071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:50.154075 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:50.154248 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:50.154981 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:50.154986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:50.155211 (+ 225us) client_negotiation.cc:769] Sending connection context
0809 04:49:50.155590 (+ 379us) client_negotiation.cc:240] Negotiation successful
0809 04:49:50.155909 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":619,"threads_started":1}
I20260809 04:49:50.156898 2126 heartbeater.cc:511] Master 127.31.93.62:44229 requested a full tablet report, sending...
I20260809 04:49:50.157055 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:50.141874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:40139 (local address 127.31.93.62:44229)
0809 04:49:50.142075 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:50.142081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:50.142873 (+ 792us) server_negotiation.cc:407] Connection header received
0809 04:49:50.143001 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:50.143008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:50.143093 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:50.143202 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:50.144059 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.145174 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.146718 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.147278 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.147411 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:50.149911 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:50.149918 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:50.149922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:50.149966 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:50.153088 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:50.153840 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:50.153844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:50.153846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:50.153960 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:50.154405 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:50.154410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:50.154413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:50.154825 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:50.154995 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:50.156657 (+ 1662us) server_negotiation.cc:299] Negotiation successful
0809 04:49:50.156824 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:49:50.158331 1560 ts_manager.cc:194] Registered new tserver with Master: 8f068481bea1413db2de885a33564966 (127.31.93.2:35339)
I20260809 04:49:50.159255 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 1903
I20260809 04:49:50.159379 1560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:42579
I20260809 04:49:50.159448 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:49:50.159930 32116 external_mini_cluster.cc:1428] Setting key cd5795035bd761ba6b6fb8ecb3015f06
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:49:50.196959 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44229
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:49:50.513190 2133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:49:50.513540 2133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:49:50.513702 2133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:49:50.522682 2133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:49:50.522822 2133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:49:50.522958 2133 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:49:50.523073 2133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:49:50.534662 2133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.93.62:44229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:49:50.536026 2133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:49:50.537582 2133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:49:50.554224 2140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:50.554337 2142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:50.554371 2144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:49:50.556151 2133 server_base.cc:1034] running on GCE node
I20260809 04:49:50.557016 2133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:49:50.557941 2133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:49:50.559234 2133 hybrid_clock.cc:648] HybridClock initialized: now 1786250990559084 us; error 43 us; skew 500 ppm
I20260809 04:49:50.559604 2133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:49:50 dist-test-slave-4tdj krb5kdc[1483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:49:50.563529 2133 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:49:50.565261 2133 webserver.cc:460] Webserver started at http://127.31.93.3:44339/ using document root <none> and password file <none>
I20260809 04:49:50.566062 2133 fs_manager.cc:359] Metadata directory not provided
I20260809 04:49:50.566188 2133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:49:50.566685 2133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:49:50.569972 2133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "1726b793396b4611b422d637773075ef"
format_stamp: "Formatted at 2026-08-09 04:49:50 on dist-test-slave-4tdj"
server_key: "685a158f219bae3c02345b254513f30f"
server_key_iv: "527c174aeb97b7ea484496212d4c408f"
server_key_version: "encryptionkey@0"
I20260809 04:49:50.570854 2133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "1726b793396b4611b422d637773075ef"
format_stamp: "Formatted at 2026-08-09 04:49:50 on dist-test-slave-4tdj"
server_key: "685a158f219bae3c02345b254513f30f"
server_key_iv: "527c174aeb97b7ea484496212d4c408f"
server_key_version: "encryptionkey@0"
I20260809 04:49:50.576510 2133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 04:49:50.580139 2159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:50.581334 2133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:49:50.581476 2133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "1726b793396b4611b422d637773075ef"
format_stamp: "Formatted at 2026-08-09 04:49:50 on dist-test-slave-4tdj"
server_key: "685a158f219bae3c02345b254513f30f"
server_key_iv: "527c174aeb97b7ea484496212d4c408f"
server_key_version: "encryptionkey@0"
I20260809 04:49:50.581691 2133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:49:50.602051 2133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:49:50.607084 2133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:49:50.607332 2133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:49:50.611101 2133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:49:50.611198 2133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:50.611469 2133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:49:50.611547 2133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:49:50 dist-test-slave-4tdj krb5kdc[1483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:50.636705 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:50.611192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44229 (local address 127.31.93.3:60827)
0809 04:49:50.612840 (+ 1648us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:50.612884 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:50.615152 (+ 2268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:50.615834 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:50.615842 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:50.616443 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:50.617022 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:50.617038 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.618043 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.618047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:50.620637 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:50.620655 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.621863 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.621872 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:50.622199 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:50.622995 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:50.623037 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:50.627058 (+ 4021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:50.631203 (+ 4145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:50.631212 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:50.631226 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:50.631823 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:50.632292 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:50.632298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:50.632302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:50.632724 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:50.633391 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:50.633399 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:50.633966 (+ 567us) client_negotiation.cc:769] Sending connection context
0809 04:49:50.634243 (+ 277us) client_negotiation.cc:240] Negotiation successful
0809 04:49:50.634944 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1420,"thread_start_us":1024,"threads_started":1}
I20260809 04:49:50.637347 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:50.610643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:60827 (local address 127.31.93.62:44229)
0809 04:49:50.610871 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:50.610877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:50.613009 (+ 2132us) server_negotiation.cc:407] Connection header received
0809 04:49:50.615423 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:50.615429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:50.615519 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:50.615649 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:50.617196 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.617898 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.620798 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.621477 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.621617 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:50.627390 (+ 5773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:50.627399 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:50.627403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:50.627448 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:50.631068 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:50.631950 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:50.631956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:50.631960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:50.632172 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:50.632847 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:50.632851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:50.632853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:50.633226 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:50.636611 (+ 3385us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:50.636777 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 04:49:50.637007 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 04:49:50.668967 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:50.647138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:34121 (local address 127.31.93.62:44229)
0809 04:49:50.647299 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:50.647304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:50.648487 (+ 1183us) server_negotiation.cc:407] Connection header received
0809 04:49:50.648722 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:50.648728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:50.648810 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:50.648898 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:50.650824 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.651833 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.654218 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.654950 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.655092 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:50.657951 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:50.657956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:50.657959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:50.657990 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:50.665704 (+ 7714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:50.666869 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:50.666873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:50.666875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:50.667023 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:50.667424 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:50.667427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:50.667429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:50.667668 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:50.667773 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:50.668401 (+ 628us) server_negotiation.cc:299] Negotiation successful
0809 04:49:50.668553 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 04:49:50.669543 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:50.647154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44229 (local address 127.31.93.3:34121)
0809 04:49:50.648371 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:50.648392 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:50.648535 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:50.648949 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:50.648954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:50.649718 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:50.650641 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:50.650657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.651984 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.651988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:50.654056 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:50.654076 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.655314 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.655323 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:50.655765 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:50.656600 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:50.656625 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:50.657749 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:50.665840 (+ 8091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:50.665845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:50.665850 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:50.666755 (+ 905us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:50.667120 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:50.667126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:50.667130 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:50.667334 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:50.667762 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:50.667767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:50.668104 (+ 337us) client_negotiation.cc:769] Sending connection context
0809 04:49:50.668270 (+ 166us) client_negotiation.cc:240] Negotiation successful
0809 04:49:50.668799 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"thread_start_us":733,"threads_started":1}
I20260809 04:49:50.669854 1898 heartbeater.cc:503] Master 127.31.93.62:44229 was elected leader, sending a full tablet report...
I20260809 04:49:50.689075 2133 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:35349
I20260809 04:49:50.689154 2362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:35349 every 8 connection(s)
I20260809 04:49:50.690382 2133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:49:50.693085 2363 heartbeater.cc:347] Connected to a master server at 127.31.93.62:44229
I20260809 04:49:50.693097 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 2133
I20260809 04:49:50.693305 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:49:50.693413 2363 heartbeater.cc:464] Registering TS with master...
I20260809 04:49:50.693997 32116 external_mini_cluster.cc:1428] Setting key 42703fa50bb18416281e710f6f39d925
I20260809 04:49:50.694808 2363 heartbeater.cc:511] Master 127.31.93.62:44229 requested a full tablet report, sending...
I20260809 04:49:50.696581 1559 ts_manager.cc:194] Registered new tserver with Master: 1726b793396b4611b422d637773075ef (127.31.93.3:35349)
I20260809 04:49:50.697489 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:49:50.697916 1559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:60827
Aug 09 04:49:50 dist-test-slave-4tdj krb5kdc[1483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250990, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 04:49:50 dist-test-slave-4tdj krb5kdc[1483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250990, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:50.749473 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:50.732935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.31.93.62:44229)
0809 04:49:50.733196 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:50.733205 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:50.733626 (+ 421us) server_negotiation.cc:407] Connection header received
0809 04:49:50.733880 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:50.733885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:50.733972 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:50.734065 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:50.736154 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.736972 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.738586 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.739005 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.739100 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:50.743426 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:50.743431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:50.743434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:50.743473 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:50.746964 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:50.747514 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:50.747518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:50.747520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:50.747748 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:50.748081 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:50.748086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:50.748089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:50.748457 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:50.748712 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:50.748899 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 04:49:50.749123 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 04:49:50.753803 1559 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:49:50.763572 1559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:49:50.791486 2386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:50.777290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:35349 (local address 127.0.0.1:36936)
0809 04:49:50.779797 (+ 2507us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:50.779847 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:50.780029 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:50.780829 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:50.780837 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:50.780910 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:50.781593 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:50.781608 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.784772 (+ 3164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.784778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:50.788017 (+ 3239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:50.788030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.790336 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.790344 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:50.790585 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:50.790712 (+ 127us) client_negotiation.cc:769] Sending connection context
0809 04:49:50.790842 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 04:49:50.791027 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2224,"thread_start_us":2059,"threads_started":1}
I20260809 04:49:50.792412 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:50.777622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.31.93.3:35349)
0809 04:49:50.779482 (+ 1860us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:50.779492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:50.780160 (+ 668us) server_negotiation.cc:407] Connection header received
0809 04:49:50.780260 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:50.780267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:50.780504 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:50.781814 (+ 1310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:50.782311 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.784599 (+ 2288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.788216 (+ 3617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.789828 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.789952 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:50.790189 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:50.791871 (+ 1682us) server_negotiation.cc:299] Negotiation successful
0809 04:49:50.792099 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1517,"thread_start_us":1250,"threads_started":1}
I20260809 04:49:50.796164 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:50.778726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56904 (local address 127.31.93.1:34927)
0809 04:49:50.780083 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:50.780091 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:50.787507 (+ 7416us) server_negotiation.cc:407] Connection header received
0809 04:49:50.787782 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:50.787790 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:50.788019 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:50.788207 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:50.789348 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.790477 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.793249 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.794983 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.795101 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:50.795300 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:50.795711 (+ 411us) server_negotiation.cc:299] Negotiation successful
0809 04:49:50.795865 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1205,"thread_start_us":1008,"threads_started":1}
I20260809 04:49:50.796454 2390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:50.777870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:34927 (local address 127.0.0.1:56904)
0809 04:49:50.787390 (+ 9520us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:50.787405 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:50.787563 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:50.788410 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:50.788415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:50.788439 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:50.789182 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:50.789197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.790602 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.790608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:50.793079 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:50.793094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.795414 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.795418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:50.795528 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:50.795551 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:49:50.795628 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 04:49:50.795756 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9352,"mutex_wait_us":19,"thread_start_us":8874,"threads_started":1}
I20260809 04:49:50.797405 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:50.780847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.31.93.2:35339)
0809 04:49:50.786831 (+ 5984us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:50.786839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:50.788291 (+ 1452us) server_negotiation.cc:407] Connection header received
0809 04:49:50.788401 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:50.788408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:50.788618 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:50.788897 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:50.789980 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.791847 (+ 1867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.794714 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.796287 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.796403 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:50.796581 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:50.797069 (+ 488us) server_negotiation.cc:299] Negotiation successful
0809 04:49:50.797199 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5808,"thread_start_us":5601,"threads_started":1}
I20260809 04:49:50.797533 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:50.780301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:35339 (local address 127.0.0.1:52146)
0809 04:49:50.788025 (+ 7724us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:50.788043 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:50.788171 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:50.788846 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:50.788851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:50.788879 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:50.789688 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:50.789704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.792016 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.792022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:50.794497 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:50.794512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:50.796711 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:50.796717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:50.796883 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:50.796915 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 04:49:50.797006 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 04:49:50.797132 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7611,"mutex_wait_us":2,"thread_start_us":6793,"threads_started":1}
I20260809 04:49:50.797318 2248 tablet_service.cc:1467] Processing CreateTablet for tablet 2dead411f5784bf9956f2bc5537a437d (DEFAULT_TABLE table=test-table [id=5c2aed817e554de89249ec3d8f4f5e6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:50.799033 2248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dead411f5784bf9956f2bc5537a437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:50.801366 2008 tablet_service.cc:1467] Processing CreateTablet for tablet 2dead411f5784bf9956f2bc5537a437d (DEFAULT_TABLE table=test-table [id=5c2aed817e554de89249ec3d8f4f5e6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:50.803026 2008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dead411f5784bf9956f2bc5537a437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:50.805126 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 2dead411f5784bf9956f2bc5537a437d (DEFAULT_TABLE table=test-table [id=5c2aed817e554de89249ec3d8f4f5e6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:50.806394 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dead411f5784bf9956f2bc5537a437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:50.817780 2393 tablet_bootstrap.cc:492] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef: Bootstrap starting.
I20260809 04:49:50.820176 2394 tablet_bootstrap.cc:492] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105: Bootstrap starting.
I20260809 04:49:50.822046 2396 tablet_bootstrap.cc:492] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966: Bootstrap starting.
I20260809 04:49:50.823172 2393 tablet_bootstrap.cc:654] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:50.823901 2394 tablet_bootstrap.cc:654] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:50.824986 2393 log.cc:824] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef: Log is configured to *not* fsync() on all Append() calls
I20260809 04:49:50.825441 2394 log.cc:824] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105: Log is configured to *not* fsync() on all Append() calls
I20260809 04:49:50.826436 2396 tablet_bootstrap.cc:654] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:50.827795 2396 log.cc:824] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966: Log is configured to *not* fsync() on all Append() calls
I20260809 04:49:50.829150 2394 tablet_bootstrap.cc:492] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105: No bootstrap required, opened a new log
I20260809 04:49:50.829170 2393 tablet_bootstrap.cc:492] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef: No bootstrap required, opened a new log
I20260809 04:49:50.829484 2393 ts_tablet_manager.cc:1397] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.005s
I20260809 04:49:50.829551 2394 ts_tablet_manager.cc:1397] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260809 04:49:50.830687 2396 tablet_bootstrap.cc:492] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966: No bootstrap required, opened a new log
I20260809 04:49:50.830926 2396 ts_tablet_manager.cc:1397] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260809 04:49:50.840942 2396 raft_consensus.cc:348] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } }
I20260809 04:49:50.841351 2396 raft_consensus.cc:374] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:50.841468 2396 raft_consensus.cc:729] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f068481bea1413db2de885a33564966, State: Initialized, Role: FOLLOWER
I20260809 04:49:50.842075 2396 consensus_queue.cc:260] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } }
I20260809 04:49:50.842422 2394 raft_consensus.cc:348] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } }
I20260809 04:49:50.842655 2393 raft_consensus.cc:348] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } }
I20260809 04:49:50.843051 2394 raft_consensus.cc:374] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:50.843245 2394 raft_consensus.cc:729] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb131a95dd3244b5a1ea0cd7aba4e105, State: Initialized, Role: FOLLOWER
I20260809 04:49:50.843379 2393 raft_consensus.cc:374] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:50.843528 2126 heartbeater.cc:503] Master 127.31.93.62:44229 was elected leader, sending a full tablet report...
I20260809 04:49:50.843657 2393 raft_consensus.cc:729] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1726b793396b4611b422d637773075ef, State: Initialized, Role: FOLLOWER
I20260809 04:49:50.844147 2396 ts_tablet_manager.cc:1428] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260809 04:49:50.844075 2394 consensus_queue.cc:260] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } }
I20260809 04:49:50.844652 2393 consensus_queue.cc:260] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } }
I20260809 04:49:50.846201 2394 ts_tablet_manager.cc:1428] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105: Time spent starting tablet: real 0.017s user 0.004s sys 0.012s
I20260809 04:49:50.851352 2363 heartbeater.cc:503] Master 127.31.93.62:44229 was elected leader, sending a full tablet report...
I20260809 04:49:50.851632 2393 ts_tablet_manager.cc:1428] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef: Time spent starting tablet: real 0.022s user 0.011s sys 0.012s
W20260809 04:49:50.906494 2127 tablet.cc:2369] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:49:50.912411 1899 tablet.cc:2369] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:49:50.942695 2365 tablet.cc:2369] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:49:51.022985 2409 raft_consensus.cc:482] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:49:51.023479 2409 raft_consensus.cc:504] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } }
I20260809 04:49:51.026197 2409 leader_election.cc:288] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1726b793396b4611b422d637773075ef (127.31.93.3:35349), 8f068481bea1413db2de885a33564966 (127.31.93.2:35339)
I20260809 04:49:51.028822 2408 raft_consensus.cc:482] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:49:51.029248 2408 raft_consensus.cc:504] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } }
I20260809 04:49:51.037417 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.026509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:55727 (local address 127.31.93.3:35349)
0809 04:49:51.027750 (+ 1241us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.027756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.027778 (+ 22us) server_negotiation.cc:407] Connection header received
0809 04:49:51.027862 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.027867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.027951 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.028047 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:51.029705 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.030786 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.034443 (+ 3657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.035810 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.035908 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.036093 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.036996 (+ 903us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.037123 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1021}
I20260809 04:49:51.038412 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.027741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:38261 (local address 127.31.93.2:35339)
0809 04:49:51.027964 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.027968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.029129 (+ 1161us) server_negotiation.cc:407] Connection header received
0809 04:49:51.029294 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.029300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.029408 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.029501 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:51.030448 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.031717 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.034867 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.036080 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.036188 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.036285 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.037908 (+ 1623us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.038032 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 04:49:51.038533 2411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.026412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:35349 (local address 127.31.93.1:55727)
0809 04:49:51.027542 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:51.027562 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:51.027707 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:51.028343 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:51.028349 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:51.028382 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:51.029565 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.029581 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.030978 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.030983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.034249 (+ 3266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.034268 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.036549 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.036557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.036798 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.036833 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 04:49:51.037851 (+ 1018us) client_negotiation.cc:240] Negotiation successful
0809 04:49:51.037976 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":22,"thread_start_us":611,"threads_started":1}
I20260809 04:49:51.039816 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dead411f5784bf9956f2bc5537a437d" candidate_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1726b793396b4611b422d637773075ef" is_pre_election: true
I20260809 04:49:51.040586 2282 raft_consensus.cc:2454] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb131a95dd3244b5a1ea0cd7aba4e105 in term 0.
I20260809 04:49:51.041667 2414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.027632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:35339 (local address 127.31.93.1:38261)
0809 04:49:51.029005 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:51.029021 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:51.029164 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:51.029559 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:51.029564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:51.029591 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:51.030247 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.030261 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.031903 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.031909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.034632 (+ 2723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.034647 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.037123 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.037132 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.037359 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.037391 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 04:49:51.038874 (+ 1483us) client_negotiation.cc:240] Negotiation successful
0809 04:49:51.039001 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"mutex_wait_us":19,"thread_start_us":910,"threads_started":1}
I20260809 04:49:51.041599 1704 leader_election.cc:302] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1726b793396b4611b422d637773075ef, cb131a95dd3244b5a1ea0cd7aba4e105; no voters:
I20260809 04:49:51.042788 2409 raft_consensus.cc:2781] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:49:51.042984 2409 raft_consensus.cc:482] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:49:51.043112 2409 raft_consensus.cc:3037] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:51.043450 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.033271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:36253 (local address 127.31.93.1:34927)
0809 04:49:51.033649 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.033655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.033673 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:49:51.033774 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.033780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.033847 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.033928 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:51.035195 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.036402 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.039781 (+ 3379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.042374 (+ 2593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.042529 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.042650 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.043084 (+ 434us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.043205 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260809 04:49:51.043664 2415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.031594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:34927 (local address 127.31.93.2:36253)
0809 04:49:51.032430 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:51.032448 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:51.032565 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:51.034163 (+ 1598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:51.034168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:51.034196 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:51.035016 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.035032 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.036583 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.036588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.039567 (+ 2979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.039583 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.042731 (+ 3148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.042736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.042926 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.042959 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 04:49:51.043049 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 04:49:51.043183 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":1898,"threads_started":1}
I20260809 04:49:51.046427 2408 leader_election.cc:288] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1726b793396b4611b422d637773075ef (127.31.93.3:35349), cb131a95dd3244b5a1ea0cd7aba4e105 (127.31.93.1:34927)
I20260809 04:49:51.047216 2409 raft_consensus.cc:504] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } }
I20260809 04:49:51.047886 2409 leader_election.cc:288] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [CANDIDATE]: Term 1 election: Requested vote from peers 1726b793396b4611b422d637773075ef (127.31.93.3:35349), 8f068481bea1413db2de885a33564966 (127.31.93.2:35339)
I20260809 04:49:51.048666 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dead411f5784bf9956f2bc5537a437d" candidate_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1726b793396b4611b422d637773075ef"
I20260809 04:49:51.048892 2282 raft_consensus.cc:3037] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:51.049753 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dead411f5784bf9956f2bc5537a437d" candidate_uuid: "8f068481bea1413db2de885a33564966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" is_pre_election: true
I20260809 04:49:51.050141 2415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.033123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:35349 (local address 127.31.93.2:56033)
0809 04:49:51.043902 (+ 10779us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:51.043919 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:51.044029 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:51.044308 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:51.044312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:51.044335 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:51.044469 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.044476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.045562 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.045568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.047736 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.047748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.049281 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.049288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.049582 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.049606 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:49:51.049688 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 04:49:51.049857 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10689,"thread_start_us":13982,"threads_started":1}
I20260809 04:49:51.050161 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.032185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:56033 (local address 127.31.93.3:35349)
0809 04:49:51.037815 (+ 5630us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.037821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.043984 (+ 6163us) server_negotiation.cc:407] Connection header received
0809 04:49:51.044141 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.044146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.044212 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.044283 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:51.044582 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.045441 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.047900 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.049049 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.049183 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.049281 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.049733 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.049854 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5532,"thread_start_us":14829,"threads_started":1}
I20260809 04:49:51.050379 1814 raft_consensus.cc:2379] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f068481bea1413db2de885a33564966 in current term 1: Already voted for candidate cb131a95dd3244b5a1ea0cd7aba4e105 in this term.
I20260809 04:49:51.051278 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dead411f5784bf9956f2bc5537a437d" candidate_uuid: "8f068481bea1413db2de885a33564966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1726b793396b4611b422d637773075ef" is_pre_election: true
I20260809 04:49:51.051755 2046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dead411f5784bf9956f2bc5537a437d" candidate_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f068481bea1413db2de885a33564966"
I20260809 04:49:51.052415 2046 raft_consensus.cc:3037] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:51.053099 2282 raft_consensus.cc:2454] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb131a95dd3244b5a1ea0cd7aba4e105 in term 1.
I20260809 04:49:51.053105 2048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dead411f5784bf9956f2bc5537a437d" candidate_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f068481bea1413db2de885a33564966" is_pre_election: true
I20260809 04:49:51.053565 1704 leader_election.cc:302] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1726b793396b4611b422d637773075ef, cb131a95dd3244b5a1ea0cd7aba4e105; no voters:
I20260809 04:49:51.053849 2409 raft_consensus.cc:2781] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:49:51.054138 1932 leader_election.cc:302] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f068481bea1413db2de885a33564966; no voters: 1726b793396b4611b422d637773075ef, cb131a95dd3244b5a1ea0cd7aba4e105
I20260809 04:49:51.054828 2409 raft_consensus.cc:686] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [term 1 LEADER]: Becoming Leader. State: Replica: cb131a95dd3244b5a1ea0cd7aba4e105, State: Running, Role: LEADER
I20260809 04:49:51.055277 2409 consensus_queue.cc:237] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } }
I20260809 04:49:51.057060 2046 raft_consensus.cc:2454] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb131a95dd3244b5a1ea0cd7aba4e105 in term 1.
I20260809 04:49:51.057363 2408 raft_consensus.cc:2726] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 04:49:51.060420 1558 catalog_manager.cc:5495] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb131a95dd3244b5a1ea0cd7aba4e105 (127.31.93.1). New cstate: current_term: 1 leader_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } health_report { overall_health: HEALTHY } } }
I20260809 04:49:51.107194 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.100164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.31.93.1:34927)
0809 04:49:51.100355 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.100361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.100675 (+ 314us) server_negotiation.cc:407] Connection header received
0809 04:49:51.100954 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.100958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.101049 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.101164 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:49:51.101568 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.102409 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.104778 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.105158 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.105253 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.105941 (+ 688us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:49:51.106501 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:49:51.106612 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.106747 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.106810 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 04:49:51.117792 2282 raft_consensus.cc:1261] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef [term 1 FOLLOWER]: Refusing update from remote peer cb131a95dd3244b5a1ea0cd7aba4e105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:49:51.118840 2409 consensus_queue.cc:1035] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:49:51.119382 2046 raft_consensus.cc:1261] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966 [term 1 FOLLOWER]: Refusing update from remote peer cb131a95dd3244b5a1ea0cd7aba4e105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:49:51.120111 2409 consensus_queue.cc:1035] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:49:51.156100 1560 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:49:51.156396 1560 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:49:51.161449 1560 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9: Sending DeleteTablet for 3 replicas of tablet 2dead411f5784bf9956f2bc5537a437d
I20260809 04:49:51.163271 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dead411f5784bf9956f2bc5537a437d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:49:51 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:56904
I20260809 04:49:51.163297 2248 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dead411f5784bf9956f2bc5537a437d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:49:51 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:36936
I20260809 04:49:51.164150 2008 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dead411f5784bf9956f2bc5537a437d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:49:51 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:52146
I20260809 04:49:51.168136 1558 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:49:51.168829 1558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:49:51.174880 2440 tablet_replica.cc:330] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef: stopping tablet replica
I20260809 04:49:51.175935 2441 tablet_replica.cc:330] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105: stopping tablet replica
I20260809 04:49:51.176018 2440 raft_consensus.cc:2229] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:49:51.176478 2440 raft_consensus.cc:2258] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:49:51.176525 2441 raft_consensus.cc:2229] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:49:51.177412 2441 raft_consensus.cc:2258] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:49:51.177673 2007 tablet_service.cc:1467] Processing CreateTablet for tablet 30200f6ef54f417196d1852eac581dd5 (DEFAULT_TABLE table=test-table [id=2529cc845b254015a846a4840f0b4ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:51.178434 2007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30200f6ef54f417196d1852eac581dd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:51.179764 2441 ts_tablet_manager.cc:1905] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:49:51.181483 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 30200f6ef54f417196d1852eac581dd5 (DEFAULT_TABLE table=test-table [id=2529cc845b254015a846a4840f0b4ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:51.181914 2440 ts_tablet_manager.cc:1905] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:49:51.181985 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30200f6ef54f417196d1852eac581dd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:51.186635 2246 tablet_service.cc:1467] Processing CreateTablet for tablet 30200f6ef54f417196d1852eac581dd5 (DEFAULT_TABLE table=test-table [id=2529cc845b254015a846a4840f0b4ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:51.187067 2246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30200f6ef54f417196d1852eac581dd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:51.190161 2442 tablet_replica.cc:330] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966: stopping tablet replica
I20260809 04:49:51.190192 2396 tablet_bootstrap.cc:492] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966: Bootstrap starting.
I20260809 04:49:51.190421 2394 tablet_bootstrap.cc:492] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105: Bootstrap starting.
I20260809 04:49:51.190739 2442 raft_consensus.cc:2229] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:49:51.192219 2394 tablet_bootstrap.cc:654] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:51.192628 2396 tablet_bootstrap.cc:654] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:51.193776 2393 tablet_bootstrap.cc:492] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef: Bootstrap starting.
I20260809 04:49:51.196041 2393 tablet_bootstrap.cc:654] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:51.196686 2441 ts_tablet_manager.cc:1918] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:49:51.197016 2441 log.cc:1196] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/2dead411f5784bf9956f2bc5537a437d
I20260809 04:49:51.198061 2441 ts_tablet_manager.cc:1939] T 2dead411f5784bf9956f2bc5537a437d P cb131a95dd3244b5a1ea0cd7aba4e105: Deleting consensus metadata
I20260809 04:49:51.200887 2440 ts_tablet_manager.cc:1918] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:49:51.201138 2440 log.cc:1196] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/2dead411f5784bf9956f2bc5537a437d
I20260809 04:49:51.200294 1537 catalog_manager.cc:4841] TS cb131a95dd3244b5a1ea0cd7aba4e105 (127.31.93.1:34927): tablet 2dead411f5784bf9956f2bc5537a437d (table test-table [id=5c2aed817e554de89249ec3d8f4f5e6c]) successfully deleted
I20260809 04:49:51.201818 2440 ts_tablet_manager.cc:1939] T 2dead411f5784bf9956f2bc5537a437d P 1726b793396b4611b422d637773075ef: Deleting consensus metadata
I20260809 04:49:51.203912 1535 catalog_manager.cc:4841] TS 1726b793396b4611b422d637773075ef (127.31.93.3:35349): tablet 2dead411f5784bf9956f2bc5537a437d (table test-table [id=5c2aed817e554de89249ec3d8f4f5e6c]) successfully deleted
I20260809 04:49:51.204357 2442 raft_consensus.cc:2258] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:49:51.206705 2442 ts_tablet_manager.cc:1905] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:49:51.206768 2393 tablet_bootstrap.cc:492] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef: No bootstrap required, opened a new log
I20260809 04:49:51.206974 2393 ts_tablet_manager.cc:1397] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20260809 04:49:51.207819 2396 tablet_bootstrap.cc:492] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966: No bootstrap required, opened a new log
I20260809 04:49:51.208056 2396 ts_tablet_manager.cc:1397] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.007s
I20260809 04:49:51.207993 2393 raft_consensus.cc:348] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.208325 2393 raft_consensus.cc:374] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:51.208557 2393 raft_consensus.cc:729] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1726b793396b4611b422d637773075ef, State: Initialized, Role: FOLLOWER
I20260809 04:49:51.208866 2393 consensus_queue.cc:260] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.209617 2393 ts_tablet_manager.cc:1428] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 04:49:51.210099 2394 tablet_bootstrap.cc:492] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105: No bootstrap required, opened a new log
I20260809 04:49:51.210021 2396 raft_consensus.cc:348] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.210369 2394 ts_tablet_manager.cc:1397] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.008s
I20260809 04:49:51.210351 2396 raft_consensus.cc:374] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:51.210584 2396 raft_consensus.cc:729] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f068481bea1413db2de885a33564966, State: Initialized, Role: FOLLOWER
I20260809 04:49:51.210870 2396 consensus_queue.cc:260] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.212184 2394 raft_consensus.cc:348] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.212534 2394 raft_consensus.cc:374] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:51.212729 2394 raft_consensus.cc:729] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb131a95dd3244b5a1ea0cd7aba4e105, State: Initialized, Role: FOLLOWER
I20260809 04:49:51.213097 2394 consensus_queue.cc:260] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.213959 2394 ts_tablet_manager.cc:1428] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 04:49:51.215267 2396 ts_tablet_manager.cc:1428] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260809 04:49:51.217429 2442 ts_tablet_manager.cc:1918] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:49:51.217687 2442 log.cc:1196] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/2dead411f5784bf9956f2bc5537a437d
I20260809 04:49:51.218530 2442 ts_tablet_manager.cc:1939] T 2dead411f5784bf9956f2bc5537a437d P 8f068481bea1413db2de885a33564966: Deleting consensus metadata
I20260809 04:49:51.220628 1535 catalog_manager.cc:4841] TS 8f068481bea1413db2de885a33564966 (127.31.93.2:35339): tablet 2dead411f5784bf9956f2bc5537a437d (table test-table [id=5c2aed817e554de89249ec3d8f4f5e6c]) successfully deleted
I20260809 04:49:51.275512 2410 raft_consensus.cc:482] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:49:51.275712 2410 raft_consensus.cc:504] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.277562 2410 leader_election.cc:288] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb131a95dd3244b5a1ea0cd7aba4e105 (127.31.93.1:34927), 8f068481bea1413db2de885a33564966 (127.31.93.2:35339)
I20260809 04:49:51.287443 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.278528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:43151 (local address 127.31.93.2:35339)
0809 04:49:51.278708 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.278714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.279163 (+ 449us) server_negotiation.cc:407] Connection header received
0809 04:49:51.279378 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.279383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.279457 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.279541 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:51.280369 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.281646 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.284501 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.286240 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.286435 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.286530 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.287018 (+ 488us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.287159 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 04:49:51.287710 2450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.277760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:35339 (local address 127.31.93.3:43151)
0809 04:49:51.279076 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:51.279093 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:51.279241 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:51.279634 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:51.279639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:51.279668 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:51.280197 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.280211 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.281758 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.281763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.284307 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.284320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.286459 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.286466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.286835 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.286873 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 04:49:51.286984 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 04:49:51.287121 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"mutex_wait_us":50,"thread_start_us":703,"threads_started":1}
I20260809 04:49:51.287895 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.278162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:60257 (local address 127.31.93.1:34927)
0809 04:49:51.278376 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.278382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.278854 (+ 472us) server_negotiation.cc:407] Connection header received
0809 04:49:51.279235 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.279242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.279327 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.279431 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:51.279996 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.281333 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.284968 (+ 3635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.286635 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.286748 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.286861 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.287400 (+ 539us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.287585 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 04:49:51.288520 2046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30200f6ef54f417196d1852eac581dd5" candidate_uuid: "1726b793396b4611b422d637773075ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f068481bea1413db2de885a33564966" is_pre_election: true
I20260809 04:49:51.288806 2448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.277733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:34927 (local address 127.31.93.3:60257)
0809 04:49:51.278734 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:51.278757 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:51.279008 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:51.279455 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:51.279459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:51.279484 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:51.279825 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.279836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.281479 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.281484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.284786 (+ 3302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.284801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.286960 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.286968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.287203 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.287236 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 04:49:51.288149 (+ 913us) client_negotiation.cc:240] Negotiation successful
0809 04:49:51.288276 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":42,"thread_start_us":1046,"threads_started":1}
I20260809 04:49:51.288795 2046 raft_consensus.cc:2454] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1726b793396b4611b422d637773075ef in term 0.
I20260809 04:49:51.289379 2161 leader_election.cc:302] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1726b793396b4611b422d637773075ef, 8f068481bea1413db2de885a33564966; no voters:
I20260809 04:49:51.289726 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30200f6ef54f417196d1852eac581dd5" candidate_uuid: "1726b793396b4611b422d637773075ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" is_pre_election: true
I20260809 04:49:51.289916 2410 raft_consensus.cc:2781] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:49:51.290127 2410 raft_consensus.cc:482] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:49:51.290122 1814 raft_consensus.cc:2454] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1726b793396b4611b422d637773075ef in term 0.
I20260809 04:49:51.290344 2410 raft_consensus.cc:3037] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:51.294636 2410 raft_consensus.cc:504] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.295579 2410 leader_election.cc:288] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [CANDIDATE]: Term 1 election: Requested vote from peers cb131a95dd3244b5a1ea0cd7aba4e105 (127.31.93.1:34927), 8f068481bea1413db2de885a33564966 (127.31.93.2:35339)
I20260809 04:49:51.295938 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30200f6ef54f417196d1852eac581dd5" candidate_uuid: "1726b793396b4611b422d637773075ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105"
I20260809 04:49:51.296128 1814 raft_consensus.cc:3037] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:51.296077 2046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30200f6ef54f417196d1852eac581dd5" candidate_uuid: "1726b793396b4611b422d637773075ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f068481bea1413db2de885a33564966"
I20260809 04:49:51.296347 2046 raft_consensus.cc:3037] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:51.299094 1814 raft_consensus.cc:2454] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1726b793396b4611b422d637773075ef in term 1.
I20260809 04:49:51.299443 2163 leader_election.cc:302] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1726b793396b4611b422d637773075ef, cb131a95dd3244b5a1ea0cd7aba4e105; no voters:
I20260809 04:49:51.299759 2410 raft_consensus.cc:2781] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:49:51.300199 2410 raft_consensus.cc:686] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [term 1 LEADER]: Becoming Leader. State: Replica: 1726b793396b4611b422d637773075ef, State: Running, Role: LEADER
I20260809 04:49:51.300311 2046 raft_consensus.cc:2454] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1726b793396b4611b422d637773075ef in term 1.
I20260809 04:49:51.300843 2410 consensus_queue.cc:237] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.304287 1558 catalog_manager.cc:5495] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 1726b793396b4611b422d637773075ef (127.31.93.3). New cstate: current_term: 1 leader_uuid: "1726b793396b4611b422d637773075ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } health_report { overall_health: HEALTHY } } }
Aug 09 04:49:51 dist-test-slave-4tdj krb5kdc[1483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250989, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:51.362696 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.345180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.31.93.62:44229)
0809 04:49:51.346538 (+ 1358us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.346542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.346562 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:49:51.346611 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.346616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.346699 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.346816 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:51.348318 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.349110 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.351169 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.351554 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.351644 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.356251 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:51.356258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:51.356262 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:51.356292 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:51.359507 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:51.360409 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:51.360412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:51.360415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:51.360593 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:51.361013 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:51.361016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:51.361018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:51.361316 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:51.361459 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.361936 (+ 477us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.362256 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":1243,"thread_start_us":1109,"threads_started":1}
I20260809 04:49:51.362857 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.345179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44229 (local address 127.0.0.1:45958)
0809 04:49:51.346027 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:51.346056 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:51.346201 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:51.346879 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:51.346885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:51.347923 (+ 1038us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:51.348182 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.348196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.349254 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.349259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.351034 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.351050 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.351765 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.351773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.351978 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.352640 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:51.352671 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:51.355967 (+ 3296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:51.359648 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:51.359661 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:51.359669 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:51.360306 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:51.360700 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:51.360707 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:51.360710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:51.360912 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:51.361454 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:51.361463 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:51.361670 (+ 207us) client_negotiation.cc:769] Sending connection context
0809 04:49:51.361858 (+ 188us) client_negotiation.cc:240] Negotiation successful
0809 04:49:51.362136 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":483,"threads_started":1}
I20260809 04:49:51.368842 1558 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:45958:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 04:49:51.370141 1558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:49:51.376945 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 0e83f5f2d9884f3ea01178146b3ef40e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3cf153d9a788412a93b09fee508b8b99]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:49:51.377034 2007 tablet_service.cc:1467] Processing CreateTablet for tablet 0e83f5f2d9884f3ea01178146b3ef40e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3cf153d9a788412a93b09fee508b8b99]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:49:51.377521 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e83f5f2d9884f3ea01178146b3ef40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:51.377682 2007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e83f5f2d9884f3ea01178146b3ef40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:51.379420 2246 tablet_service.cc:1467] Processing CreateTablet for tablet 0e83f5f2d9884f3ea01178146b3ef40e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3cf153d9a788412a93b09fee508b8b99]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:49:51.379972 2246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e83f5f2d9884f3ea01178146b3ef40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:51.387259 2394 tablet_bootstrap.cc:492] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105: Bootstrap starting.
I20260809 04:49:51.387382 2396 tablet_bootstrap.cc:492] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966: Bootstrap starting.
I20260809 04:49:51.389026 2393 tablet_bootstrap.cc:492] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef: Bootstrap starting.
I20260809 04:49:51.389745 2396 tablet_bootstrap.cc:654] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:51.389839 2394 tablet_bootstrap.cc:654] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:51.391539 2393 tablet_bootstrap.cc:654] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:51.394060 2396 tablet_bootstrap.cc:492] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966: No bootstrap required, opened a new log
I20260809 04:49:51.394385 2396 ts_tablet_manager.cc:1397] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260809 04:49:51.395323 2393 tablet_bootstrap.cc:492] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef: No bootstrap required, opened a new log
I20260809 04:49:51.395448 2396 raft_consensus.cc:348] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.395646 2393 ts_tablet_manager.cc:1397] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260809 04:49:51.395728 2396 raft_consensus.cc:374] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:51.395848 2396 raft_consensus.cc:729] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f068481bea1413db2de885a33564966, State: Initialized, Role: FOLLOWER
I20260809 04:49:51.396245 2396 consensus_queue.cc:260] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.396565 2393 raft_consensus.cc:348] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.396831 2393 raft_consensus.cc:374] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:51.396912 2393 raft_consensus.cc:729] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1726b793396b4611b422d637773075ef, State: Initialized, Role: FOLLOWER
I20260809 04:49:51.396772 2408 tablet_replica.cc:439] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } }
I20260809 04:49:51.397074 2408 tablet_replica.cc:442] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:49:51.397640 2396 ts_tablet_manager.cc:1428] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 04:49:51.398066 2393 consensus_queue.cc:260] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.398590 2410 tablet_replica.cc:439] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } }
I20260809 04:49:51.398962 2410 tablet_replica.cc:442] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:49:51.399601 2393 ts_tablet_manager.cc:1428] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260809 04:49:51.400171 2394 tablet_bootstrap.cc:492] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105: No bootstrap required, opened a new log
I20260809 04:49:51.400345 2394 ts_tablet_manager.cc:1397] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20260809 04:49:51.402326 2394 raft_consensus.cc:348] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.402683 2394 raft_consensus.cc:374] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:51.402794 2394 raft_consensus.cc:729] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb131a95dd3244b5a1ea0cd7aba4e105, State: Initialized, Role: FOLLOWER
I20260809 04:49:51.403126 2394 consensus_queue.cc:260] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.403757 2433 tablet_replica.cc:439] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } }
I20260809 04:49:51.404471 2433 tablet_replica.cc:442] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:49:51.404565 2394 ts_tablet_manager.cc:1428] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260809 04:49:51.412382 2433 raft_consensus.cc:482] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:49:51.412593 2433 raft_consensus.cc:504] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.413467 2433 leader_election.cc:288] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f068481bea1413db2de885a33564966 (127.31.93.2:35339), 1726b793396b4611b422d637773075ef (127.31.93.3:35349)
I20260809 04:49:51.414000 2046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e83f5f2d9884f3ea01178146b3ef40e" candidate_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f068481bea1413db2de885a33564966" is_pre_election: true
I20260809 04:49:51.414067 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e83f5f2d9884f3ea01178146b3ef40e" candidate_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1726b793396b4611b422d637773075ef" is_pre_election: true
I20260809 04:49:51.414263 2282 raft_consensus.cc:2454] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb131a95dd3244b5a1ea0cd7aba4e105 in term 0.
I20260809 04:49:51.414321 2046 raft_consensus.cc:2454] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb131a95dd3244b5a1ea0cd7aba4e105 in term 0.
I20260809 04:49:51.414798 1704 leader_election.cc:302] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1726b793396b4611b422d637773075ef, cb131a95dd3244b5a1ea0cd7aba4e105; no voters:
I20260809 04:49:51.415181 2433 raft_consensus.cc:2781] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:49:51.415402 2433 raft_consensus.cc:482] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:49:51.415550 2433 raft_consensus.cc:3037] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:51.418236 2433 raft_consensus.cc:504] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.418923 2433 leader_election.cc:288] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [CANDIDATE]: Term 1 election: Requested vote from peers 8f068481bea1413db2de885a33564966 (127.31.93.2:35339), 1726b793396b4611b422d637773075ef (127.31.93.3:35349)
I20260809 04:49:51.419350 2046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e83f5f2d9884f3ea01178146b3ef40e" candidate_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f068481bea1413db2de885a33564966"
I20260809 04:49:51.419472 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e83f5f2d9884f3ea01178146b3ef40e" candidate_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1726b793396b4611b422d637773075ef"
I20260809 04:49:51.419684 2282 raft_consensus.cc:3037] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:51.419689 2046 raft_consensus.cc:3037] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:51.422896 2282 raft_consensus.cc:2454] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb131a95dd3244b5a1ea0cd7aba4e105 in term 1.
I20260809 04:49:51.422935 2046 raft_consensus.cc:2454] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb131a95dd3244b5a1ea0cd7aba4e105 in term 1.
I20260809 04:49:51.423348 1704 leader_election.cc:302] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f068481bea1413db2de885a33564966, cb131a95dd3244b5a1ea0cd7aba4e105; no voters:
I20260809 04:49:51.423794 2433 raft_consensus.cc:2781] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:49:51.424038 2433 raft_consensus.cc:686] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [term 1 LEADER]: Becoming Leader. State: Replica: cb131a95dd3244b5a1ea0cd7aba4e105, State: Running, Role: LEADER
I20260809 04:49:51.424446 2433 consensus_queue.cc:237] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } }
I20260809 04:49:51.428136 2422 tablet_replica.cc:439] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105: TxnStatusTablet state changed. Reason: New leader cb131a95dd3244b5a1ea0cd7aba4e105. Latest consensus state: current_term: 1 leader_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } }
I20260809 04:49:51.428674 2422 tablet_replica.cc:442] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:49:51.429405 1558 catalog_manager.cc:5495] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb131a95dd3244b5a1ea0cd7aba4e105 (127.31.93.1). New cstate: current_term: 1 leader_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } health_report { overall_health: UNKNOWN } } }
I20260809 04:49:51.430269 2478 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 04:49:51.430647 2478 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 04:49:51.454002 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.448123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:34927 (local address 127.0.0.1:56914)
0809 04:49:51.448327 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:51.448347 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:51.448514 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:51.448889 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:51.448892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:51.448903 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:49:51.449086 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.449094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.450574 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.450577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.451657 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.451670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.452675 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.452679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.452972 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.453030 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:49:51.453444 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:49:51.453455 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 04:49:51.453518 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 04:49:51.453659 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 04:49:51.454034 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.447888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.31.93.1:34927)
0809 04:49:51.448081 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.448087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.448431 (+ 344us) server_negotiation.cc:407] Connection header received
0809 04:49:51.448678 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.448684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.448773 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.448879 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:49:51.449213 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.450446 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.451803 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.452551 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.452673 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.453218 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:49:51.453354 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:49:51.453447 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.453579 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.453660 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 04:49:51.458315 2046 raft_consensus.cc:1261] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966 [term 1 FOLLOWER]: Refusing update from remote peer cb131a95dd3244b5a1ea0cd7aba4e105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:49:51.458405 2282 raft_consensus.cc:1261] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef [term 1 FOLLOWER]: Refusing update from remote peer cb131a95dd3244b5a1ea0cd7aba4e105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:49:51.459012 2422 consensus_queue.cc:1035] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:49:51.459425 2433 consensus_queue.cc:1035] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:49:51.462546 2410 tablet_replica.cc:439] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef: TxnStatusTablet state changed. Reason: New leader cb131a95dd3244b5a1ea0cd7aba4e105. Latest consensus state: current_term: 1 leader_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } }
I20260809 04:49:51.462953 2410 tablet_replica.cc:442] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:49:51.464870 2408 tablet_replica.cc:439] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966: TxnStatusTablet state changed. Reason: New leader cb131a95dd3244b5a1ea0cd7aba4e105. Latest consensus state: current_term: 1 leader_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } }
I20260809 04:49:51.465162 2408 tablet_replica.cc:442] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:49:51.465023 2422 tablet_replica.cc:439] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } }
I20260809 04:49:51.465327 2422 tablet_replica.cc:442] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:49:51.468622 2433 tablet_replica.cc:439] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } }
I20260809 04:49:51.469058 2433 tablet_replica.cc:442] T 0e83f5f2d9884f3ea01178146b3ef40e P cb131a95dd3244b5a1ea0cd7aba4e105: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:49:51.470719 2408 tablet_replica.cc:439] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } }
I20260809 04:49:51.470901 2408 tablet_replica.cc:442] T 0e83f5f2d9884f3ea01178146b3ef40e P 8f068481bea1413db2de885a33564966: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:49:51.475255 2410 tablet_replica.cc:439] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 } } peers { permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 } } peers { permanent_uuid: "1726b793396b4611b422d637773075ef" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 35349 } } }
I20260809 04:49:51.475577 2410 tablet_replica.cc:442] T 0e83f5f2d9884f3ea01178146b3ef40e P 1726b793396b4611b422d637773075ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:49:51.484665 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.478162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.31.93.3:35349)
0809 04:49:51.478351 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.478356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.478375 (+ 19us) server_negotiation.cc:407] Connection header received
0809 04:49:51.478465 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.478470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.478582 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.478734 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:49:51.479304 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.480126 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.481991 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.482610 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.482725 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.483441 (+ 716us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:49:51.484033 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:49:51.484161 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.484269 (+ 108us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.484347 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 04:49:51.504930 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.497115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:34927 (local address 127.31.93.3:36417)
0809 04:49:51.498184 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:51.498208 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:51.498350 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:51.498703 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:51.498706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:51.498714 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:49:51.499028 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.499038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.500487 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.500491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.502215 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.502228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.503570 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.503576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.503795 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.503848 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:49:51.504340 (+ 492us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:49:51.504347 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 04:49:51.504397 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 04:49:51.504494 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":737,"threads_started":1}
I20260809 04:49:51.505025 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.497115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:36417 (local address 127.31.93.1:34927)
0809 04:49:51.497311 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.497318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.498317 (+ 999us) server_negotiation.cc:407] Connection header received
0809 04:49:51.498482 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.498489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.498591 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.498695 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:49:51.499194 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.500369 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.502378 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.503261 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.503433 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.504086 (+ 653us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:49:51.504239 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:49:51.504350 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.504456 (+ 106us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.504550 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 04:49:51.515779 2046 raft_consensus.cc:1261] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966 [term 1 FOLLOWER]: Refusing update from remote peer 1726b793396b4611b422d637773075ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:49:51.516283 1814 raft_consensus.cc:1261] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105 [term 1 FOLLOWER]: Refusing update from remote peer 1726b793396b4611b422d637773075ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:49:51.517164 2410 consensus_queue.cc:1035] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f068481bea1413db2de885a33564966" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:49:51.517506 2455 consensus_queue.cc:1035] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb131a95dd3244b5a1ea0cd7aba4e105" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 34927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:49:51.523636 2428 mvcc.cc:205] Tried to move back new op lower bound from 7316484061242040320 to 7316484060372533248. Current Snapshot: MvccSnapshot[applied={T|T < 7316484061242040320}]
I20260809 04:49:51.562183 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.552870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:35349 (local address 127.31.93.1:42189)
0809 04:49:51.554943 (+ 2073us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:51.554963 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:51.555157 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:51.555475 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:51.555480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:51.555492 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:49:51.555978 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.555994 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.557394 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.557400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.559630 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:51.559647 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.560549 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.560556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:51.560783 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.560861 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:49:51.561319 (+ 458us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:49:51.561331 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 04:49:51.561435 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 04:49:51.561621 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1874,"thread_start_us":1737,"threads_started":1}
I20260809 04:49:51.562783 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.552870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:42189 (local address 127.31.93.3:35349)
0809 04:49:51.553123 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.553129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.555041 (+ 1912us) server_negotiation.cc:407] Connection header received
0809 04:49:51.555301 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.555304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.555367 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.555428 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:49:51.556217 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.557241 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.559771 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.560353 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.560449 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.561080 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:49:51.561181 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:49:51.561263 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.561483 (+ 220us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.561582 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260809 04:49:51.600473 1558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:49:51.600795 1558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:49:51.604789 1558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a6bea19994647a583bc844a43a18af9: Sending DeleteTablet for 3 replicas of tablet 30200f6ef54f417196d1852eac581dd5
I20260809 04:49:51.606088 2007 tablet_service.cc:1514] Processing DeleteTablet for tablet 30200f6ef54f417196d1852eac581dd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:49:51 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:52146
I20260809 04:49:51.607549 2442 tablet_replica.cc:330] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966: stopping tablet replica
I20260809 04:49:51.607851 2442 raft_consensus.cc:2229] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:49:51.608067 2442 raft_consensus.cc:2258] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:49:51.608593 2246 tablet_service.cc:1514] Processing DeleteTablet for tablet 30200f6ef54f417196d1852eac581dd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:49:51 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:36936
I20260809 04:49:51.608922 2440 tablet_replica.cc:330] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef: stopping tablet replica
I20260809 04:49:51.609145 2440 raft_consensus.cc:2229] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:49:51.609728 2440 raft_consensus.cc:2258] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:49:51.610963 2442 ts_tablet_manager.cc:1905] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:49:51.614827 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 30200f6ef54f417196d1852eac581dd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:49:51 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:56904
I20260809 04:49:51.615131 2441 tablet_replica.cc:330] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105: stopping tablet replica
I20260809 04:49:51.615334 2441 raft_consensus.cc:2229] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:49:51.615550 2441 raft_consensus.cc:2258] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:49:51.617625 2440 ts_tablet_manager.cc:1905] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:49:51.622731 2442 ts_tablet_manager.cc:1918] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:49:51.622963 2441 ts_tablet_manager.cc:1905] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:49:51.623044 2442 log.cc:1196] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/30200f6ef54f417196d1852eac581dd5
I20260809 04:49:51.623795 2442 ts_tablet_manager.cc:1939] T 30200f6ef54f417196d1852eac581dd5 P 8f068481bea1413db2de885a33564966: Deleting consensus metadata
I20260809 04:49:51.624223 2440 ts_tablet_manager.cc:1918] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:49:51.624701 2440 log.cc:1196] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/30200f6ef54f417196d1852eac581dd5
I20260809 04:49:51.625119 1535 catalog_manager.cc:4841] TS 8f068481bea1413db2de885a33564966 (127.31.93.2:35339): tablet 30200f6ef54f417196d1852eac581dd5 (table test-table [id=2529cc845b254015a846a4840f0b4ea4]) successfully deleted
I20260809 04:49:51.625764 2440 ts_tablet_manager.cc:1939] T 30200f6ef54f417196d1852eac581dd5 P 1726b793396b4611b422d637773075ef: Deleting consensus metadata
I20260809 04:49:51.627998 1535 catalog_manager.cc:4841] TS 1726b793396b4611b422d637773075ef (127.31.93.3:35349): tablet 30200f6ef54f417196d1852eac581dd5 (table test-table [id=2529cc845b254015a846a4840f0b4ea4]) successfully deleted
I20260809 04:49:51.632777 2441 ts_tablet_manager.cc:1918] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:49:51.632994 2441 log.cc:1196] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/30200f6ef54f417196d1852eac581dd5
I20260809 04:49:51.633553 2441 ts_tablet_manager.cc:1939] T 30200f6ef54f417196d1852eac581dd5 P cb131a95dd3244b5a1ea0cd7aba4e105: Deleting consensus metadata
I20260809 04:49:51.634703 1537 catalog_manager.cc:4841] TS cb131a95dd3244b5a1ea0cd7aba4e105 (127.31.93.1:34927): tablet 30200f6ef54f417196d1852eac581dd5 (table test-table [id=2529cc845b254015a846a4840f0b4ea4]) successfully deleted
Aug 09 04:49:51 dist-test-slave-4tdj krb5kdc[1483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250990, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:49:51.644403 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.615687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56918 (local address 127.31.93.1:34927)
0809 04:49:51.615926 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.615932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.617645 (+ 1713us) server_negotiation.cc:407] Connection header received
0809 04:49:51.617966 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.617971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.622758 (+ 4787us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.622858 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:51.627424 (+ 4566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.628774 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.630748 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.631504 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.631621 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.637282 (+ 5661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:51.637293 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:51.637307 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:51.637341 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:51.641232 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:51.641998 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:51.642013 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:51.642015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:51.642179 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:51.642634 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:51.642638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:51.642640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:51.642998 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:51.643439 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.643680 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.643922 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
W20260809 04:49:51.645314 1884 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56918
I20260809 04:49:51.673991 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.655413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45960 (local address 127.31.93.62:44229)
0809 04:49:51.655788 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.655794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.656269 (+ 475us) server_negotiation.cc:407] Connection header received
0809 04:49:51.656418 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.656423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.656494 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.656583 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:51.658207 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.659110 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.661429 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.662429 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.662530 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.666556 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:51.666564 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:51.666569 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:51.666608 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:51.670785 (+ 4177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:51.671729 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:51.671735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:51.671739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:51.671870 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:51.672282 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:51.672289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:51.672293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:51.672652 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:51.672861 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.673399 (+ 538us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.673630 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
W20260809 04:49:51.675365 1558 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45960
I20260809 04:49:51.677675 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 1684
I20260809 04:49:51.704795 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.680317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.31.93.1:34927)
0809 04:49:51.680473 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.680478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.686784 (+ 6306us) server_negotiation.cc:407] Connection header received
0809 04:49:51.686923 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.686929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.686998 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.687077 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:51.690039 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.691200 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.693011 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.693721 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.693811 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.697050 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:51.697057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:51.697062 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:51.697102 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:51.701414 (+ 4312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:51.702239 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:51.702244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:51.702248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:51.702429 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:51.702832 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:51.702837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:51.702841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:51.703189 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:51.703488 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.704050 (+ 562us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.704332 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260809 04:49:51.706346 1884 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56920
W20260809 04:49:51.706925 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 1684: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:49:51.707095 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 1684
I20260809 04:49:51.737036 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 1903
Aug 09 04:49:51 dist-test-slave-4tdj krb5kdc[1483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250990, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:49:51.751757 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.737767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.31.93.2:35339)
0809 04:49:51.737951 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.737955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.738169 (+ 214us) server_negotiation.cc:407] Connection header received
0809 04:49:51.738365 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.738371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.738441 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.738680 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:51.739832 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.740621 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.741930 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.742398 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.742512 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.746205 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:51.746216 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:51.746229 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:51.746261 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:51.749191 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:51.749784 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:51.749792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:51.749794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:51.749921 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:51.750270 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:51.750275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:51.750278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:51.750656 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:51.750925 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.751218 (+ 293us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.751415 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260809 04:49:51.752343 2119 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52158
W20260809 04:49:51.752914 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 1903: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:49:51.753067 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 1903
I20260809 04:49:51.789593 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 2133
Aug 09 04:49:51 dist-test-slave-4tdj krb5kdc[1483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250990, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:49:51.804600 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:51.790357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.31.93.3:35349)
0809 04:49:51.790529 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:51.790535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:51.790616 (+ 81us) server_negotiation.cc:407] Connection header received
0809 04:49:51.790806 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:51.790812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:51.790880 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:51.790977 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:51.792173 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.792924 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.794090 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:51.794628 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:51.794755 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:51.798923 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:51.798940 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:51.798956 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:51.798990 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:51.801898 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:51.802714 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:51.802722 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:51.802724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:51.802872 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:51.803251 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:51.803255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:51.803257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:51.803590 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:51.803831 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:51.804133 (+ 302us) server_negotiation.cc:299] Negotiation successful
0809 04:49:51.804294 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260809 04:49:51.805284 2357 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36964
W20260809 04:49:51.805869 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 2133: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:49:51.805996 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 2133
I20260809 04:49:51.824716 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 1502
I20260809 04:49:51.825693 1629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:49:51.936337 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 1502
2026-08-09T04:49:51Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4589 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:49:52 dist-test-slave-4tdj krb5kdc[2511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:49:52 dist-test-slave-4tdj krb5kdc[2511](info): set up 2 sockets
Aug 09 04:49:52 dist-test-slave-4tdj krb5kdc[2511](info): commencing operation
krb5kdc: starting...
W20260809 04:49:53.139837 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:49:53 dist-test-slave-4tdj krb5kdc[2511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:49:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:49:53.311867 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:42363
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:44001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:42363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:49:53.632774 2533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:49:53.633042 2533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:49:53.633167 2533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:49:53.640136 2533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:49:53.640233 2533 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 04:49:53.640300 2533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:49:53.640378 2533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:49:53.640440 2533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:49:53.640550 2533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:49:53.650043 2533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:44001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:42363
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:42363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:49:53.651227 2533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:49:53.652487 2533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:49:53.663519 2545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:53.663618 2538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:53.663625 2543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:49:53.664384 2533 server_base.cc:1034] running on GCE node
I20260809 04:49:53.665118 2533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:49:53.666433 2533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:49:53.667644 2533 hybrid_clock.cc:648] HybridClock initialized: now 1786250993667582 us; error 40 us; skew 500 ppm
I20260809 04:49:53.667948 2533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:49:53 dist-test-slave-4tdj krb5kdc[2511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250993, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:49:53.670933 2533 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:49:53.672245 2533 webserver.cc:460] Webserver started at http://127.31.93.62:42521/ using document root <none> and password file <none>
I20260809 04:49:53.672916 2533 fs_manager.cc:359] Metadata directory not provided
I20260809 04:49:53.673017 2533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:49:53.673306 2533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:49:53.676158 2533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "6b1bf0d5a1e4431499b0d7e0e2a8364c"
format_stamp: "Formatted at 2026-08-09 04:49:53 on dist-test-slave-4tdj"
server_key: "05024d0daa3adda7317fddf2e56c00a5"
server_key_iv: "8a0705e5787b097134c46598d133a47b"
server_key_version: "encryptionkey@0"
I20260809 04:49:53.676891 2533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "6b1bf0d5a1e4431499b0d7e0e2a8364c"
format_stamp: "Formatted at 2026-08-09 04:49:53 on dist-test-slave-4tdj"
server_key: "05024d0daa3adda7317fddf2e56c00a5"
server_key_iv: "8a0705e5787b097134c46598d133a47b"
server_key_version: "encryptionkey@0"
I20260809 04:49:53.681685 2533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260809 04:49:53.684934 2555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:53.686159 2533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 04:49:53.686343 2533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "6b1bf0d5a1e4431499b0d7e0e2a8364c"
format_stamp: "Formatted at 2026-08-09 04:49:53 on dist-test-slave-4tdj"
server_key: "05024d0daa3adda7317fddf2e56c00a5"
server_key_iv: "8a0705e5787b097134c46598d133a47b"
server_key_version: "encryptionkey@0"
I20260809 04:49:53.686547 2533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:49:53.696779 2533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:49:53.700634 2533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:49:53.700862 2533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:49:53.720578 2533 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:42363
I20260809 04:49:53.720613 2674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:42363 every 8 connection(s)
I20260809 04:49:53.721869 2533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:49:53.725431 2675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:53.729280 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 2533
I20260809 04:49:53.729471 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:49:53.730103 32116 external_mini_cluster.cc:1428] Setting key 2f2867278010f78d1b55f7d8cf462a8f
I20260809 04:49:53.736017 2675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c: Bootstrap starting.
I20260809 04:49:53.739619 2675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:53.740769 2675 log.cc:824] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c: Log is configured to *not* fsync() on all Append() calls
I20260809 04:49:53.744028 2675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c: No bootstrap required, opened a new log
Aug 09 04:49:53 dist-test-slave-4tdj krb5kdc[2511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:53.753651 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:53.731969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57340 (local address 127.31.93.62:42363)
0809 04:49:53.732785 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:53.732796 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:53.732839 (+ 43us) server_negotiation.cc:407] Connection header received
0809 04:49:53.734769 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:53.734792 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:53.735169 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:53.735470 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:53.737226 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:53.738576 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:53.740237 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:53.740945 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:53.741078 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:53.745620 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:53.745631 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:53.745652 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:53.745710 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:53.749449 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:53.750093 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:53.750106 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:53.750114 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:53.750343 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:53.750667 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:53.750676 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:53.750679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:53.751352 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:53.751670 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:53.751940 (+ 270us) server_negotiation.cc:299] Negotiation successful
0809 04:49:53.752371 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":346,"threads_started":1}
I20260809 04:49:53.754009 2675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1bf0d5a1e4431499b0d7e0e2a8364c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 42363 } }
I20260809 04:49:53.754384 2675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:53.754487 2675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1bf0d5a1e4431499b0d7e0e2a8364c, State: Initialized, Role: FOLLOWER
I20260809 04:49:53.755090 2675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1bf0d5a1e4431499b0d7e0e2a8364c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 42363 } }
I20260809 04:49:53.755329 2675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:49:53.755463 2675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:49:53.755625 2675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:53.757701 2675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1bf0d5a1e4431499b0d7e0e2a8364c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 42363 } }
I20260809 04:49:53.758157 2675 leader_election.cc:302] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b1bf0d5a1e4431499b0d7e0e2a8364c; no voters:
I20260809 04:49:53.758690 2675 leader_election.cc:288] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:49:53.758800 2683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:49:53.759435 2683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [term 1 LEADER]: Becoming Leader. State: Replica: 6b1bf0d5a1e4431499b0d7e0e2a8364c, State: Running, Role: LEADER
I20260809 04:49:53.760003 2683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1bf0d5a1e4431499b0d7e0e2a8364c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 42363 } }
I20260809 04:49:53.760479 2675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:49:53.762671 2685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b1bf0d5a1e4431499b0d7e0e2a8364c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1bf0d5a1e4431499b0d7e0e2a8364c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 42363 } } }
I20260809 04:49:53.763024 2685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [sys.catalog]: This master's current role is: LEADER
I20260809 04:49:53.763720 2686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b1bf0d5a1e4431499b0d7e0e2a8364c. Latest consensus state: current_term: 1 leader_uuid: "6b1bf0d5a1e4431499b0d7e0e2a8364c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1bf0d5a1e4431499b0d7e0e2a8364c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 42363 } } }
I20260809 04:49:53.763917 2700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:49:53.763981 2686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c [sys.catalog]: This master's current role is: LEADER
I20260809 04:49:53.769946 2700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:49:53.779201 2700 catalog_manager.cc:1347] Generated new cluster ID: e47877a6a3644f54bb0fe75d4b5256db
I20260809 04:49:53.779307 2700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:49:53.814072 2700 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:49:53.815375 2700 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:49:53.830381 2700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c: Generated new TSK 0
I20260809 04:49:53.831176 2700 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:49:53.875742 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:42363
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:44001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:49:54.186247 2724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:49:54.186538 2724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:49:54.186678 2724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:49:54.193611 2724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:49:54.193717 2724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:49:54.193837 2724 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 04:49:54.193903 2724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:49:54.194015 2724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:49:54.203056 2724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:44001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.93.62:42363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:49:54.204304 2724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:49:54.205612 2724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:49:54.217669 2739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:54.217711 2737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:54.217737 2736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:49:54.218606 2724 server_base.cc:1034] running on GCE node
I20260809 04:49:54.219174 2724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:49:54.220032 2724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:49:54.221230 2724 hybrid_clock.cc:648] HybridClock initialized: now 1786250994221183 us; error 34 us; skew 500 ppm
I20260809 04:49:54.221525 2724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:49:54 dist-test-slave-4tdj krb5kdc[2511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250994, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:49:54.224366 2724 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:49:54.225677 2724 webserver.cc:460] Webserver started at http://127.31.93.1:33199/ using document root <none> and password file <none>
I20260809 04:49:54.226328 2724 fs_manager.cc:359] Metadata directory not provided
I20260809 04:49:54.226452 2724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:49:54.226747 2724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:49:54.229452 2724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "4c0c20ce539b4721a5d4c2a543cd227a"
format_stamp: "Formatted at 2026-08-09 04:49:54 on dist-test-slave-4tdj"
server_key: "34d0f2809263ed60a27fb12a1fbb797c"
server_key_iv: "f5b868f44743a90bba08f4a306882ebb"
server_key_version: "encryptionkey@0"
I20260809 04:49:54.230170 2724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "4c0c20ce539b4721a5d4c2a543cd227a"
format_stamp: "Formatted at 2026-08-09 04:49:54 on dist-test-slave-4tdj"
server_key: "34d0f2809263ed60a27fb12a1fbb797c"
server_key_iv: "f5b868f44743a90bba08f4a306882ebb"
server_key_version: "encryptionkey@0"
I20260809 04:49:54.234902 2724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 04:49:54.238036 2751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:54.239259 2724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 04:49:54.239406 2724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "4c0c20ce539b4721a5d4c2a543cd227a"
format_stamp: "Formatted at 2026-08-09 04:49:54 on dist-test-slave-4tdj"
server_key: "34d0f2809263ed60a27fb12a1fbb797c"
server_key_iv: "f5b868f44743a90bba08f4a306882ebb"
server_key_version: "encryptionkey@0"
I20260809 04:49:54.239593 2724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:49:54.256665 2724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:49:54.260247 2724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:49:54.260510 2724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:49:54.262943 2724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:49:54.263046 2724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:54.263155 2724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:49:54.263229 2724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:49:54 dist-test-slave-4tdj krb5kdc[2511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250994, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:54.284706 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:54.263615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:53603 (local address 127.31.93.62:42363)
0809 04:49:54.265826 (+ 2211us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:54.265832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:54.265853 (+ 21us) server_negotiation.cc:407] Connection header received
0809 04:49:54.267560 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:54.267566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:54.267664 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:54.267762 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:54.269150 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:54.270125 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:54.273051 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:54.273600 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:54.273734 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:54.278131 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:54.278136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:54.278140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:54.278182 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:54.281226 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:54.282194 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:54.282199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:54.282203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:54.282379 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:54.282868 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:54.282875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:54.282879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:54.283272 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:54.283463 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:54.284062 (+ 599us) server_negotiation.cc:299] Negotiation successful
0809 04:49:54.284305 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2092,"thread_start_us":1991,"threads_started":1}
I20260809 04:49:54.285341 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:54.264093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:42363 (local address 127.31.93.1:53603)
0809 04:49:54.265102 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:54.265165 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:54.267309 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:54.267991 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:54.267999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:54.268464 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:54.268985 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:54.269002 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:54.270396 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:54.270403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:54.272876 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:54.272892 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:54.273877 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:54.273884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:54.274190 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:54.274914 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:54.274945 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:54.277908 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:54.281396 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:54.281404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:54.281419 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:54.282032 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:54.282516 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:54.282520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:54.282522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:54.282734 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:54.283417 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:54.283423 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:54.283732 (+ 309us) client_negotiation.cc:769] Sending connection context
0809 04:49:54.283932 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 04:49:54.284268 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":481,"threads_started":1}
I20260809 04:49:54.306926 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:54.292237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:42363 (local address 127.31.93.1:49977)
0809 04:49:54.293221 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:54.293242 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:54.293369 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:54.293709 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:54.293714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:54.294208 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:54.294749 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:54.294763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:54.296041 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:54.296046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:54.297540 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:54.297555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:54.298513 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:54.298520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:54.298700 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:54.299316 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:54.299340 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:54.300252 (+ 912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:54.303970 (+ 3718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:54.303975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:54.303980 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:54.304613 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:54.305044 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:54.305049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:54.305053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:54.305236 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:54.305931 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:54.305937 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:54.306172 (+ 235us) client_negotiation.cc:769] Sending connection context
0809 04:49:54.306344 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 04:49:54.306591 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":394,"threads_started":1}
I20260809 04:49:54.307009 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:54.292583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:49977 (local address 127.31.93.62:42363)
0809 04:49:54.293158 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:54.293164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:54.293320 (+ 156us) server_negotiation.cc:407] Connection header received
0809 04:49:54.293493 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:54.293499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:54.293574 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:54.293664 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:54.294893 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:54.295918 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:54.297681 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:54.298341 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:54.298461 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:54.300427 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:54.300433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:54.300438 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:54.300476 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:54.303823 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:54.304741 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:54.304747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:54.304750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:54.304917 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:54.305402 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:54.305408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:54.305412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:54.305795 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:54.305955 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:54.306446 (+ 491us) server_negotiation.cc:299] Negotiation successful
0809 04:49:54.306686 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451}
I20260809 04:49:54.314002 2724 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:35287
I20260809 04:49:54.314051 2957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:35287 every 8 connection(s)
I20260809 04:49:54.315213 2724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:49:54.317178 2958 heartbeater.cc:347] Connected to a master server at 127.31.93.62:42363
I20260809 04:49:54.317373 2958 heartbeater.cc:464] Registering TS with master...
I20260809 04:49:54.317973 2958 heartbeater.cc:511] Master 127.31.93.62:42363 requested a full tablet report, sending...
I20260809 04:49:54.319831 2586 ts_manager.cc:194] Registered new tserver with Master: 4c0c20ce539b4721a5d4c2a543cd227a (127.31.93.1:35287)
I20260809 04:49:54.321269 2586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:53603
I20260809 04:49:54.322120 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 2724
I20260809 04:49:54.322386 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:49:54.323033 32116 external_mini_cluster.cc:1428] Setting key 1efad8aab849c74a88559b0035915356
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:49:54.361135 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:42363
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:44001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:49:54.666406 2964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:49:54.666684 2964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:49:54.666831 2964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:49:54.673565 2964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:49:54.673676 2964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:49:54.673801 2964 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 04:49:54.673866 2964 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:49:54.673974 2964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:49:54.682988 2964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:44001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.93.62:42363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:49:54.684104 2964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:49:54.685364 2964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:49:54.697396 2973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:54.697338 2978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:54.697430 2975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:49:54.698127 2964 server_base.cc:1034] running on GCE node
I20260809 04:49:54.698735 2964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:49:54.699537 2964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:49:54.700726 2964 hybrid_clock.cc:648] HybridClock initialized: now 1786250994700681 us; error 48 us; skew 500 ppm
I20260809 04:49:54.700992 2964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:49:54 dist-test-slave-4tdj krb5kdc[2511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250994, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:49:54.703842 2964 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:49:54.705180 2964 webserver.cc:460] Webserver started at http://127.31.93.2:34845/ using document root <none> and password file <none>
I20260809 04:49:54.705796 2964 fs_manager.cc:359] Metadata directory not provided
I20260809 04:49:54.705909 2964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:49:54.706192 2964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:49:54.708937 2964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "5ec148d0a5f64c39a0dc36d55284f190"
format_stamp: "Formatted at 2026-08-09 04:49:54 on dist-test-slave-4tdj"
server_key: "64b57e52815dad0cae718e6a27f0b736"
server_key_iv: "bd26e62d82b841d6b1cc6852e6750983"
server_key_version: "encryptionkey@0"
I20260809 04:49:54.709672 2964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "5ec148d0a5f64c39a0dc36d55284f190"
format_stamp: "Formatted at 2026-08-09 04:49:54 on dist-test-slave-4tdj"
server_key: "64b57e52815dad0cae718e6a27f0b736"
server_key_iv: "bd26e62d82b841d6b1cc6852e6750983"
server_key_version: "encryptionkey@0"
I20260809 04:49:54.714524 2964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260809 04:49:54.717805 2991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:54.719072 2964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:49:54.719221 2964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "5ec148d0a5f64c39a0dc36d55284f190"
format_stamp: "Formatted at 2026-08-09 04:49:54 on dist-test-slave-4tdj"
server_key: "64b57e52815dad0cae718e6a27f0b736"
server_key_iv: "bd26e62d82b841d6b1cc6852e6750983"
server_key_version: "encryptionkey@0"
I20260809 04:49:54.719398 2964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:49:54.762439 2964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:49:54.766062 2964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:49:54.766353 2964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:49:54.768913 2964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:49:54.769029 2964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:54.769141 2964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:49:54.769225 2964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:49:54 dist-test-slave-4tdj krb5kdc[2511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250994, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:54.789844 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:54.769543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:56953 (local address 127.31.93.62:42363)
0809 04:49:54.769735 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:54.769741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:54.771109 (+ 1368us) server_negotiation.cc:407] Connection header received
0809 04:49:54.773138 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:54.773143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:54.773261 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:54.773367 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:54.775677 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:54.776798 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:54.779345 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:54.779763 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:54.779875 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:54.783470 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:54.783474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:54.783478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:54.783512 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:54.786594 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:54.787384 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:54.787390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:54.787394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:54.787513 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:54.788014 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:54.788021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:54.788024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:54.788420 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:54.788600 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:54.789276 (+ 676us) server_negotiation.cc:299] Negotiation successful
0809 04:49:54.789491 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 04:49:54.791208 3001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:54.770148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:42363 (local address 127.31.93.2:56953)
0809 04:49:54.770956 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:54.771001 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:54.772877 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:54.773657 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:54.773669 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:54.774513 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:54.775341 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:54.775482 (+ 141us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:54.777000 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:54.777010 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:54.779212 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:54.779227 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:54.779986 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:54.779994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:54.780195 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:54.780815 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:54.780835 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:54.783237 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:54.786764 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:54.786773 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:54.786807 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:54.787270 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:54.787614 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:54.787619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:54.787623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:54.787837 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:54.788547 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:54.788552 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:54.788858 (+ 306us) client_negotiation.cc:769] Sending connection context
0809 04:49:54.789141 (+ 283us) client_negotiation.cc:240] Negotiation successful
0809 04:49:54.789672 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":353,"threads_started":1}
I20260809 04:49:54.811081 3077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:54.796635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:42363 (local address 127.31.93.2:34103)
0809 04:49:54.797300 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:54.797321 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:54.797457 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:54.797718 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:54.797723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:54.798281 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:54.798734 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:54.798750 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:54.799906 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:54.799912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:54.801782 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:54.801798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:54.802648 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:54.802656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:54.802886 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:54.803548 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:54.803610 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:54.804790 (+ 1180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:54.808520 (+ 3730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:54.808523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:54.808526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:54.808886 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:54.809300 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:54.809306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:54.809310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:54.809482 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:54.810186 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:54.810190 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:54.810383 (+ 193us) client_negotiation.cc:769] Sending connection context
0809 04:49:54.810514 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 04:49:54.810760 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":332,"threads_started":1}
I20260809 04:49:54.811187 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:54.796715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:34103 (local address 127.31.93.62:42363)
0809 04:49:54.796905 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:54.796912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:54.797404 (+ 492us) server_negotiation.cc:407] Connection header received
0809 04:49:54.797548 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:54.797552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:54.797605 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:54.797680 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:54.798882 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:54.799735 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:54.801935 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:54.802461 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:54.802584 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:54.805030 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:54.805036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:54.805041 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:54.805088 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:54.808399 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:54.808980 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:54.808985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:54.808989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:54.809154 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:54.809634 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:54.809640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:54.809644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:54.810056 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:54.810211 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:54.810632 (+ 421us) server_negotiation.cc:299] Negotiation successful
0809 04:49:54.810849 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:49:54.819551 2964 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:37635
I20260809 04:49:54.819583 3197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:37635 every 8 connection(s)
I20260809 04:49:54.820664 2964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:49:54.822372 3198 heartbeater.cc:347] Connected to a master server at 127.31.93.62:42363
I20260809 04:49:54.822571 3198 heartbeater.cc:464] Registering TS with master...
I20260809 04:49:54.823156 3198 heartbeater.cc:511] Master 127.31.93.62:42363 requested a full tablet report, sending...
I20260809 04:49:54.824374 2586 ts_manager.cc:194] Registered new tserver with Master: 5ec148d0a5f64c39a0dc36d55284f190 (127.31.93.2:37635)
I20260809 04:49:54.825192 2586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:56953
I20260809 04:49:54.826479 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 2964
I20260809 04:49:54.826674 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:49:54.827224 32116 external_mini_cluster.cc:1428] Setting key 4e9f5478ab778726845ba4400dda9d1c
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:49:54.865054 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:42363
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:44001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:49:55.174649 3204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:49:55.174896 3204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:49:55.175026 3204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:49:55.181898 3204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:49:55.181993 3204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:49:55.182125 3204 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 04:49:55.182186 3204 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:49:55.182286 3204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:49:55.191589 3204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:44001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.93.62:42363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:49:55.192736 3204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:49:55.193961 3204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:49:55.206036 3211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:55.206066 3210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:49:55.206096 3213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:49:55.211583 3204 server_base.cc:1034] running on GCE node
I20260809 04:49:55.212134 3204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:49:55.212958 3204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:49:55.214147 3204 hybrid_clock.cc:648] HybridClock initialized: now 1786250995214096 us; error 45 us; skew 500 ppm
I20260809 04:49:55.214455 3204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:49:55 dist-test-slave-4tdj krb5kdc[2511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250995, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:49:55.220834 3204 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:49:55.222215 3204 webserver.cc:460] Webserver started at http://127.31.93.3:33115/ using document root <none> and password file <none>
I20260809 04:49:55.222901 3204 fs_manager.cc:359] Metadata directory not provided
I20260809 04:49:55.223016 3204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:49:55.223302 3204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:49:55.226146 3204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "eafba8ec391942baabc3df18ac19ebb7"
format_stamp: "Formatted at 2026-08-09 04:49:55 on dist-test-slave-4tdj"
server_key: "4363c90ca21b2fcba01629967abd041f"
server_key_iv: "d042fb4c0f1d9a10034106ea6381e7b4"
server_key_version: "encryptionkey@0"
I20260809 04:49:55.226933 3204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "eafba8ec391942baabc3df18ac19ebb7"
format_stamp: "Formatted at 2026-08-09 04:49:55 on dist-test-slave-4tdj"
server_key: "4363c90ca21b2fcba01629967abd041f"
server_key_iv: "d042fb4c0f1d9a10034106ea6381e7b4"
server_key_version: "encryptionkey@0"
I20260809 04:49:55.231652 3204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260809 04:49:55.234825 3223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:55.236013 3204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:49:55.236156 3204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "eafba8ec391942baabc3df18ac19ebb7"
format_stamp: "Formatted at 2026-08-09 04:49:55 on dist-test-slave-4tdj"
server_key: "4363c90ca21b2fcba01629967abd041f"
server_key_iv: "d042fb4c0f1d9a10034106ea6381e7b4"
server_key_version: "encryptionkey@0"
I20260809 04:49:55.236328 3204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:49:55.260927 3204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:49:55.264398 3204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:49:55.264650 3204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:49:55.267190 3204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:49:55.267288 3204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:49:55.267400 3204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:49:55.267485 3204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:49:55 dist-test-slave-4tdj krb5kdc[2511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250995, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:55.287670 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.267606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:37571 (local address 127.31.93.62:42363)
0809 04:49:55.267802 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:55.267808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:55.269075 (+ 1267us) server_negotiation.cc:407] Connection header received
0809 04:49:55.271110 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:55.271115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:55.271206 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:55.271333 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:55.273400 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.274458 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.276016 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.276433 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.276543 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.280560 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:55.280567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:55.280570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:55.280604 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:55.284183 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:55.285026 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:55.285032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:55.285036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:55.285210 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:55.285806 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:55.285811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:55.285814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:55.286158 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:55.286324 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:55.287075 (+ 751us) server_negotiation.cc:299] Negotiation successful
0809 04:49:55.287304 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 04:49:55.288573 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.268097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:42363 (local address 127.31.93.3:37571)
0809 04:49:55.268908 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:55.268962 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:55.270846 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:55.271601 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:55.271611 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:55.272318 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:55.273203 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.273243 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.274629 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.274634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.275872 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.275885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.276657 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.276665 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.276877 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.277500 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:55.277536 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:55.280355 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:55.284313 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:55.284320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:55.284330 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:55.284916 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:55.285352 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:55.285357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:55.285367 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:55.285679 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:55.286287 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:55.286324 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:55.286630 (+ 306us) client_negotiation.cc:769] Sending connection context
0809 04:49:55.286920 (+ 290us) client_negotiation.cc:240] Negotiation successful
0809 04:49:55.287422 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":339,"threads_started":1}
I20260809 04:49:55.309227 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.294686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:55351 (local address 127.31.93.62:42363)
0809 04:49:55.294816 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:55.294820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:55.295423 (+ 603us) server_negotiation.cc:407] Connection header received
0809 04:49:55.295609 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:55.295615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:55.295689 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:55.295786 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:55.297320 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.298433 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.300628 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.301259 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.301357 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.303841 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:55.303847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:55.303852 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:55.303889 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:55.306814 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:55.307445 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:55.307451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:55.307454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:55.307573 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:55.307901 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:55.307906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:55.307909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:55.308175 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:55.308308 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:55.308700 (+ 392us) server_negotiation.cc:299] Negotiation successful
0809 04:49:55.308896 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 04:49:55.309316 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.294733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:42363 (local address 127.31.93.3:55351)
0809 04:49:55.295322 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:55.295341 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:55.295475 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:55.295950 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:55.295959 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:55.296667 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:55.297130 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.297148 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.298630 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.298635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.300443 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.300459 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.301648 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.301656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.301896 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.302585 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:55.302605 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:55.303607 (+ 1002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:55.306940 (+ 3333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:55.306949 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:55.306953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:55.307338 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:55.307675 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:55.307679 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:55.307683 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:55.307803 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:55.308289 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:55.308294 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:55.308461 (+ 167us) client_negotiation.cc:769] Sending connection context
0809 04:49:55.308576 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 04:49:55.308879 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":332,"threads_started":1}
I20260809 04:49:55.318785 3204 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:40111
I20260809 04:49:55.318819 3437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:40111 every 8 connection(s)
I20260809 04:49:55.319859 3204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:49:55.321625 3438 heartbeater.cc:347] Connected to a master server at 127.31.93.62:42363
I20260809 04:49:55.321837 3438 heartbeater.cc:464] Registering TS with master...
I20260809 04:49:55.322465 3438 heartbeater.cc:511] Master 127.31.93.62:42363 requested a full tablet report, sending...
I20260809 04:49:55.323661 2586 ts_manager.cc:194] Registered new tserver with Master: eafba8ec391942baabc3df18ac19ebb7 (127.31.93.3:40111)
I20260809 04:49:55.323985 2958 heartbeater.cc:503] Master 127.31.93.62:42363 was elected leader, sending a full tablet report...
I20260809 04:49:55.324592 2586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:37571
I20260809 04:49:55.330021 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 3204
I20260809 04:49:55.330179 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:49:55.330627 32116 external_mini_cluster.cc:1428] Setting key 6949e326883105e18a3c03bc50972e35
I20260809 04:49:55.333141 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 04:49:55 dist-test-slave-4tdj krb5kdc[2511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 04:49:55 dist-test-slave-4tdj krb5kdc[2511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:55.378495 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.366142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.31.93.62:42363)
0809 04:49:55.366414 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:55.366420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:55.366631 (+ 211us) server_negotiation.cc:407] Connection header received
0809 04:49:55.366820 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:55.366824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:55.366890 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:55.366964 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:55.368206 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.368903 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.370019 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.370481 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.370572 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.374042 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:55.374047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:55.374051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:55.374084 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:55.376505 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:55.376996 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:55.377000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:55.377002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:55.377117 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:55.377393 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:55.377396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:55.377398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:55.377641 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:55.377833 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:55.378007 (+ 174us) server_negotiation.cc:299] Negotiation successful
0809 04:49:55.378167 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 04:49:55.382230 2585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:49:55.388972 2585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:49:55.422690 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.401795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.31.93.1:35287)
0809 04:49:55.410836 (+ 9041us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:55.410845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:55.414936 (+ 4091us) server_negotiation.cc:407] Connection header received
0809 04:49:55.415034 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:55.415042 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:55.415266 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:55.415440 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:55.416366 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.417802 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.420193 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.421199 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.421316 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.421517 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:55.422211 (+ 694us) server_negotiation.cc:299] Negotiation successful
0809 04:49:55.422387 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8868,"thread_start_us":8732,"threads_started":1}
I20260809 04:49:55.422832 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.400805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:35287 (local address 127.0.0.1:36102)
0809 04:49:55.414680 (+ 13875us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:55.414699 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:55.414813 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:55.415654 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:55.415658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:55.415682 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:55.416160 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.416174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.417959 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.417964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.420064 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.420074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.421682 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.421690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.421944 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.422030 (+ 86us) client_negotiation.cc:769] Sending connection context
0809 04:49:55.422147 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 04:49:55.422359 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13733,"mutex_wait_us":77,"spinlock_wait_cycles":1792,"thread_start_us":13724,"threads_started":1}
I20260809 04:49:55.422957 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.401225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44236 (local address 127.31.93.2:37635)
0809 04:49:55.410389 (+ 9164us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:55.410399 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:55.413890 (+ 3491us) server_negotiation.cc:407] Connection header received
0809 04:49:55.414558 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:55.414567 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:55.414820 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:55.414994 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:55.415708 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.417133 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.419858 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.421084 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.421210 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.421422 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:55.422521 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0809 04:49:55.422699 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8960,"thread_start_us":8820,"threads_started":1}
I20260809 04:49:55.423070 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.400504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:37635 (local address 127.0.0.1:44236)
0809 04:49:55.413666 (+ 13162us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:55.413702 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:55.413877 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:55.415179 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:55.415186 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:55.415225 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:55.415542 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.415551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.417270 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.417275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.419700 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.419714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.421595 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.421606 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.421901 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.422032 (+ 131us) client_negotiation.cc:769] Sending connection context
0809 04:49:55.422150 (+ 118us) client_negotiation.cc:240] Negotiation successful
0809 04:49:55.422359 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12925,"mutex_wait_us":88,"thread_start_us":14379,"threads_started":1}
I20260809 04:49:55.423254 3459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.401092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:40111 (local address 127.0.0.1:38954)
0809 04:49:55.415132 (+ 14040us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:55.415148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:55.415258 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:55.415821 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:55.415828 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:55.415854 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:55.416293 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.416307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.417922 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.417927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.419547 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.419560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.421640 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.421648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.421940 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.422030 (+ 90us) client_negotiation.cc:769] Sending connection context
0809 04:49:55.422160 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 04:49:55.422381 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13929,"mutex_wait_us":162,"thread_start_us":12213,"threads_started":1}
I20260809 04:49:55.423858 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.401445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38954 (local address 127.31.93.3:40111)
0809 04:49:55.414422 (+ 12977us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:55.414430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:55.415231 (+ 801us) server_negotiation.cc:407] Connection header received
0809 04:49:55.415434 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:55.415442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:55.415637 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:55.415826 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:55.416516 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.417784 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.419698 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.421084 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.421222 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.421439 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:55.423466 (+ 2027us) server_negotiation.cc:299] Negotiation successful
0809 04:49:55.423634 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12794,"thread_start_us":12597,"threads_started":1}
I20260809 04:49:55.426710 3068 tablet_service.cc:1467] Processing CreateTablet for tablet 1160e45e410b4ab69c80a74d91448b01 (DEFAULT_TABLE table=test-table [id=1f01630e94404595b02540c17f9e4c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:55.426913 2843 tablet_service.cc:1467] Processing CreateTablet for tablet 1160e45e410b4ab69c80a74d91448b01 (DEFAULT_TABLE table=test-table [id=1f01630e94404595b02540c17f9e4c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:55.427253 3322 tablet_service.cc:1467] Processing CreateTablet for tablet 1160e45e410b4ab69c80a74d91448b01 (DEFAULT_TABLE table=test-table [id=1f01630e94404595b02540c17f9e4c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:55.428177 2843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1160e45e410b4ab69c80a74d91448b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:55.428190 3068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1160e45e410b4ab69c80a74d91448b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:55.428496 3322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1160e45e410b4ab69c80a74d91448b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:55.438344 3466 tablet_bootstrap.cc:492] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a: Bootstrap starting.
I20260809 04:49:55.441556 3466 tablet_bootstrap.cc:654] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:55.441931 3468 tablet_bootstrap.cc:492] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7: Bootstrap starting.
I20260809 04:49:55.443076 3467 tablet_bootstrap.cc:492] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190: Bootstrap starting.
I20260809 04:49:55.443527 3466 log.cc:824] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a: Log is configured to *not* fsync() on all Append() calls
I20260809 04:49:55.446159 3468 tablet_bootstrap.cc:654] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:55.446413 3467 tablet_bootstrap.cc:654] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:55.446566 3466 tablet_bootstrap.cc:492] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a: No bootstrap required, opened a new log
I20260809 04:49:55.446836 3466 ts_tablet_manager.cc:1397] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260809 04:49:55.447194 3468 log.cc:824] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7: Log is configured to *not* fsync() on all Append() calls
I20260809 04:49:55.447576 3467 log.cc:824] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190: Log is configured to *not* fsync() on all Append() calls
I20260809 04:49:55.449893 3467 tablet_bootstrap.cc:492] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190: No bootstrap required, opened a new log
I20260809 04:49:55.449975 3468 tablet_bootstrap.cc:492] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7: No bootstrap required, opened a new log
I20260809 04:49:55.450137 3467 ts_tablet_manager.cc:1397] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260809 04:49:55.450227 3468 ts_tablet_manager.cc:1397] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260809 04:49:55.456667 3466 raft_consensus.cc:348] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:55.457067 3466 raft_consensus.cc:374] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:55.457180 3466 raft_consensus.cc:729] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0c20ce539b4721a5d4c2a543cd227a, State: Initialized, Role: FOLLOWER
I20260809 04:49:55.457120 3467 raft_consensus.cc:348] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:55.457474 3467 raft_consensus.cc:374] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:55.457568 3467 raft_consensus.cc:729] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec148d0a5f64c39a0dc36d55284f190, State: Initialized, Role: FOLLOWER
I20260809 04:49:55.457870 3466 consensus_queue.cc:260] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:55.458123 3467 consensus_queue.cc:260] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:55.460188 3468 raft_consensus.cc:348] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:55.460587 3468 raft_consensus.cc:374] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:55.460717 3468 raft_consensus.cc:729] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafba8ec391942baabc3df18ac19ebb7, State: Initialized, Role: FOLLOWER
I20260809 04:49:55.461687 3468 consensus_queue.cc:260] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:55.462845 3438 heartbeater.cc:503] Master 127.31.93.62:42363 was elected leader, sending a full tablet report...
I20260809 04:49:55.463456 3468 ts_tablet_manager.cc:1428] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20260809 04:49:55.465049 3198 heartbeater.cc:503] Master 127.31.93.62:42363 was elected leader, sending a full tablet report...
I20260809 04:49:55.465927 3467 ts_tablet_manager.cc:1428] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260809 04:49:55.468583 3466 ts_tablet_manager.cc:1428] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a: Time spent starting tablet: real 0.022s user 0.012s sys 0.006s
W20260809 04:49:55.567286 2959 tablet.cc:2369] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:49:55.571210 3440 tablet.cc:2369] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:49:55.572382 3199 tablet.cc:2369] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:49:55.698419 3472 raft_consensus.cc:482] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:49:55.698750 3472 raft_consensus.cc:504] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:55.700405 3472 leader_election.cc:288] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ec148d0a5f64c39a0dc36d55284f190 (127.31.93.2:37635), eafba8ec391942baabc3df18ac19ebb7 (127.31.93.3:40111)
I20260809 04:49:55.706928 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.700745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:35983 (local address 127.31.93.2:37635)
0809 04:49:55.700957 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:55.700963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:55.701835 (+ 872us) server_negotiation.cc:407] Connection header received
0809 04:49:55.701984 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:55.701988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:55.702048 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:55.702129 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:55.702835 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.703550 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.705244 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.705976 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.706097 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.706212 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:55.706553 (+ 341us) server_negotiation.cc:299] Negotiation successful
0809 04:49:55.706677 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 04:49:55.706977 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.700890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:43711 (local address 127.31.93.3:40111)
0809 04:49:55.701092 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:55.701098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:55.701782 (+ 684us) server_negotiation.cc:407] Connection header received
0809 04:49:55.701956 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:55.701961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:55.702028 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:55.702092 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:55.702761 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.703458 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.705230 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.705992 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.706106 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.706203 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:55.706622 (+ 419us) server_negotiation.cc:299] Negotiation successful
0809 04:49:55.706737 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 04:49:55.707028 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.700858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:37635 (local address 127.31.93.1:35983)
0809 04:49:55.701724 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:55.701739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:55.701862 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:55.702161 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:55.702165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:55.702188 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:55.702682 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.702692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.703688 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.703694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.705082 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.705094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.706180 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.706186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.706396 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.706421 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 04:49:55.706499 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 04:49:55.706616 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":34,"thread_start_us":476,"threads_started":1}
I20260809 04:49:55.707283 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.700844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:40111 (local address 127.31.93.1:43711)
0809 04:49:55.701680 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:55.701695 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:55.701821 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:55.702134 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:55.702139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:55.702162 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:55.702621 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.702630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.703583 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.703588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.705075 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.705087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.706360 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.706364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.706472 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.706493 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 04:49:55.706583 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 04:49:55.706698 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":19,"thread_start_us":530,"threads_started":1}
I20260809 04:49:55.708060 3106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1160e45e410b4ab69c80a74d91448b01" candidate_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec148d0a5f64c39a0dc36d55284f190" is_pre_election: true
I20260809 04:49:55.708227 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1160e45e410b4ab69c80a74d91448b01" candidate_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafba8ec391942baabc3df18ac19ebb7" is_pre_election: true
I20260809 04:49:55.708529 3106 raft_consensus.cc:2454] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c0c20ce539b4721a5d4c2a543cd227a in term 0.
I20260809 04:49:55.708701 3360 raft_consensus.cc:2454] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c0c20ce539b4721a5d4c2a543cd227a in term 0.
I20260809 04:49:55.709164 2753 leader_election.cc:302] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c0c20ce539b4721a5d4c2a543cd227a, 5ec148d0a5f64c39a0dc36d55284f190; no voters:
I20260809 04:49:55.709527 3472 raft_consensus.cc:2781] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:49:55.709647 3472 raft_consensus.cc:482] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:49:55.709753 3472 raft_consensus.cc:3037] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:55.711887 3472 raft_consensus.cc:504] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:55.712419 3472 leader_election.cc:288] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [CANDIDATE]: Term 1 election: Requested vote from peers 5ec148d0a5f64c39a0dc36d55284f190 (127.31.93.2:37635), eafba8ec391942baabc3df18ac19ebb7 (127.31.93.3:40111)
I20260809 04:49:55.712802 3106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1160e45e410b4ab69c80a74d91448b01" candidate_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec148d0a5f64c39a0dc36d55284f190"
I20260809 04:49:55.712834 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1160e45e410b4ab69c80a74d91448b01" candidate_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafba8ec391942baabc3df18ac19ebb7"
I20260809 04:49:55.712996 3106 raft_consensus.cc:3037] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:55.712996 3360 raft_consensus.cc:3037] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:55.716012 3106 raft_consensus.cc:2454] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c0c20ce539b4721a5d4c2a543cd227a in term 1.
I20260809 04:49:55.716012 3360 raft_consensus.cc:2454] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c0c20ce539b4721a5d4c2a543cd227a in term 1.
I20260809 04:49:55.716342 2753 leader_election.cc:302] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c0c20ce539b4721a5d4c2a543cd227a, 5ec148d0a5f64c39a0dc36d55284f190; no voters:
I20260809 04:49:55.716611 3472 raft_consensus.cc:2781] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:49:55.717000 3472 raft_consensus.cc:686] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 LEADER]: Becoming Leader. State: Replica: 4c0c20ce539b4721a5d4c2a543cd227a, State: Running, Role: LEADER
I20260809 04:49:55.717456 3472 consensus_queue.cc:237] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:55.721612 2586 catalog_manager.cc:5495] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c0c20ce539b4721a5d4c2a543cd227a (127.31.93.1). New cstate: current_term: 1 leader_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } health_report { overall_health: UNKNOWN } } }
Aug 09 04:49:55 dist-test-slave-4tdj krb5kdc[2511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250993, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:55.800910 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.788578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.31.93.62:42363)
0809 04:49:55.788740 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:55.788744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:55.789298 (+ 554us) server_negotiation.cc:407] Connection header received
0809 04:49:55.789451 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:55.789455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:55.789507 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:55.789583 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:55.790887 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.791628 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.792832 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.793245 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.793354 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.796344 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:55.796349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:55.796352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:55.796379 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:55.798493 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:55.799118 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:55.799122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:55.799124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:55.799227 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:55.799547 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:55.799551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:55.799554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:55.799813 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:55.799964 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:55.800340 (+ 376us) server_negotiation.cc:299] Negotiation successful
0809 04:49:55.800619 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":53}
I20260809 04:49:55.800931 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.788607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:42363 (local address 127.0.0.1:57354)
0809 04:49:55.789197 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:55.789217 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:55.789349 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:55.789637 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:55.789642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:55.790486 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:49:55.790754 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.790764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.791743 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.791749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.792705 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.792718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.793417 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.793423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.793556 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.794114 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:49:55.794132 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:49:55.796187 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:49:55.798608 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:55.798620 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:55.798627 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:55.799007 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:55.799321 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:49:55.799326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:49:55.799330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:49:55.799452 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:49:55.799934 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:49:55.799941 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:49:55.800105 (+ 164us) client_negotiation.cc:769] Sending connection context
0809 04:49:55.800246 (+ 141us) client_negotiation.cc:240] Negotiation successful
0809 04:49:55.800488 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":324,"threads_started":1}
I20260809 04:49:55.805506 2586 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:57354:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 04:49:55.806533 2586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:49:55.812713 2843 tablet_service.cc:1467] Processing CreateTablet for tablet 4e023dab31014ef3aeb23714b6b0672a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eb5477835ad145448c01ff3a5b926d9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:49:55.812971 3322 tablet_service.cc:1467] Processing CreateTablet for tablet 4e023dab31014ef3aeb23714b6b0672a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eb5477835ad145448c01ff3a5b926d9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:49:55.813592 2843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e023dab31014ef3aeb23714b6b0672a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:55.813709 3322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e023dab31014ef3aeb23714b6b0672a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:55.814730 3068 tablet_service.cc:1467] Processing CreateTablet for tablet 4e023dab31014ef3aeb23714b6b0672a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eb5477835ad145448c01ff3a5b926d9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:49:55.815502 3068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e023dab31014ef3aeb23714b6b0672a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:55.819069 3468 tablet_bootstrap.cc:492] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7: Bootstrap starting.
I20260809 04:49:55.820664 3468 tablet_bootstrap.cc:654] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:55.821806 3467 tablet_bootstrap.cc:492] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190: Bootstrap starting.
I20260809 04:49:55.823683 3467 tablet_bootstrap.cc:654] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:55.825709 3466 tablet_bootstrap.cc:492] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a: Bootstrap starting.
I20260809 04:49:55.827198 3466 tablet_bootstrap.cc:654] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:55.833333 3467 tablet_bootstrap.cc:492] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190: No bootstrap required, opened a new log
I20260809 04:49:55.833556 3467 ts_tablet_manager.cc:1397] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260809 04:49:55.834437 3467 raft_consensus.cc:348] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:55.834797 3467 raft_consensus.cc:374] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:55.834961 3467 raft_consensus.cc:729] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec148d0a5f64c39a0dc36d55284f190, State: Initialized, Role: FOLLOWER
I20260809 04:49:55.835259 3467 consensus_queue.cc:260] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:55.836024 3467 ts_tablet_manager.cc:1428] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 04:49:55.835821 3474 tablet_replica.cc:439] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } } }
I20260809 04:49:55.836244 3474 tablet_replica.cc:442] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:49:55.837869 3466 tablet_bootstrap.cc:492] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a: No bootstrap required, opened a new log
I20260809 04:49:55.838020 3466 ts_tablet_manager.cc:1397] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260809 04:49:55.838362 3468 tablet_bootstrap.cc:492] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7: No bootstrap required, opened a new log
I20260809 04:49:55.838836 3466 raft_consensus.cc:348] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:55.838975 3468 ts_tablet_manager.cc:1397] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.012s
I20260809 04:49:55.839071 3466 raft_consensus.cc:374] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:55.839190 3466 raft_consensus.cc:729] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0c20ce539b4721a5d4c2a543cd227a, State: Initialized, Role: FOLLOWER
I20260809 04:49:55.839445 3466 consensus_queue.cc:260] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:55.839742 3468 raft_consensus.cc:348] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:55.839847 3472 tablet_replica.cc:439] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } } }
I20260809 04:49:55.840049 3468 raft_consensus.cc:374] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:55.840124 3472 tablet_replica.cc:442] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:49:55.840162 3468 raft_consensus.cc:729] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafba8ec391942baabc3df18ac19ebb7, State: Initialized, Role: FOLLOWER
I20260809 04:49:55.840375 3468 consensus_queue.cc:260] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:55.841012 3468 ts_tablet_manager.cc:1428] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 04:49:55.840855 3476 tablet_replica.cc:439] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } } }
I20260809 04:49:55.841159 3476 tablet_replica.cc:442] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:49:55.841566 3466 ts_tablet_manager.cc:1428] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 04:49:55.991904 3476 raft_consensus.cc:482] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:49:55.992125 3476 raft_consensus.cc:504] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:55.993723 3476 leader_election.cc:288] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c0c20ce539b4721a5d4c2a543cd227a (127.31.93.1:35287), 5ec148d0a5f64c39a0dc36d55284f190 (127.31.93.2:37635)
I20260809 04:49:56.000200 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.994041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:41115 (local address 127.31.93.1:35287)
0809 04:49:55.994841 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:55.994847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:55.994868 (+ 21us) server_negotiation.cc:407] Connection header received
0809 04:49:55.994961 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:55.994966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:55.995043 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:55.995160 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:55.995685 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.996541 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.998332 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.999219 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.999319 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.999393 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:55.999758 (+ 365us) server_negotiation.cc:299] Negotiation successful
0809 04:49:55.999886 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":603,"threads_started":1}
I20260809 04:49:56.000221 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.993922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:35287 (local address 127.31.93.3:41115)
0809 04:49:55.994648 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:55.994670 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:55.994835 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:55.995178 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:55.995184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:55.995248 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:55.995517 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.995529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.996686 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.996692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.998152 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.998167 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.999385 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.999391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.999585 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.999626 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 04:49:55.999704 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 04:49:55.999801 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":22,"spinlock_wait_cycles":45952,"thread_start_us":433,"threads_started":1}
I20260809 04:49:56.000312 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.994146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:34641 (local address 127.31.93.2:37635)
0809 04:49:55.994404 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:55.994410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:55.995210 (+ 800us) server_negotiation.cc:407] Connection header received
0809 04:49:55.995398 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:55.995403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:55.995480 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:55.995576 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:49:55.996105 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.996891 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.998650 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.999386 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.999499 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.999589 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:55.999932 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 04:49:56.000055 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 04:49:56.000555 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:55.993983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:37635 (local address 127.31.93.3:34641)
0809 04:49:55.995099 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:55.995123 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:55.995263 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:55.995589 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:55.995592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:55.995614 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:49:55.995949 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.995960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.997028 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.997032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.998501 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:55.998515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:55.999587 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:55.999592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:55.999762 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:55.999794 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 04:49:55.999885 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 04:49:56.000009 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"mutex_wait_us":35,"spinlock_wait_cycles":7296,"thread_start_us":829,"threads_started":1}
I20260809 04:49:56.001034 2889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e023dab31014ef3aeb23714b6b0672a" candidate_uuid: "eafba8ec391942baabc3df18ac19ebb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" is_pre_election: true
I20260809 04:49:56.001148 3106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e023dab31014ef3aeb23714b6b0672a" candidate_uuid: "eafba8ec391942baabc3df18ac19ebb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec148d0a5f64c39a0dc36d55284f190" is_pre_election: true
I20260809 04:49:56.001384 3106 raft_consensus.cc:2454] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eafba8ec391942baabc3df18ac19ebb7 in term 0.
I20260809 04:49:56.001390 2889 raft_consensus.cc:2454] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eafba8ec391942baabc3df18ac19ebb7 in term 0.
I20260809 04:49:56.001865 3225 leader_election.cc:302] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4c0c20ce539b4721a5d4c2a543cd227a, 5ec148d0a5f64c39a0dc36d55284f190, eafba8ec391942baabc3df18ac19ebb7; no voters:
I20260809 04:49:56.002342 3476 raft_consensus.cc:2781] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:49:56.002462 3476 raft_consensus.cc:482] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:49:56.002550 3476 raft_consensus.cc:3037] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:56.004599 3476 raft_consensus.cc:504] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:56.005136 3476 leader_election.cc:288] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [CANDIDATE]: Term 1 election: Requested vote from peers 4c0c20ce539b4721a5d4c2a543cd227a (127.31.93.1:35287), 5ec148d0a5f64c39a0dc36d55284f190 (127.31.93.2:37635)
I20260809 04:49:56.005546 2889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e023dab31014ef3aeb23714b6b0672a" candidate_uuid: "eafba8ec391942baabc3df18ac19ebb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c0c20ce539b4721a5d4c2a543cd227a"
I20260809 04:49:56.005596 3106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e023dab31014ef3aeb23714b6b0672a" candidate_uuid: "eafba8ec391942baabc3df18ac19ebb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec148d0a5f64c39a0dc36d55284f190"
I20260809 04:49:56.005719 2889 raft_consensus.cc:3037] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:56.005784 3106 raft_consensus.cc:3037] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:56.007738 3106 raft_consensus.cc:2454] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafba8ec391942baabc3df18ac19ebb7 in term 1.
I20260809 04:49:56.007761 2889 raft_consensus.cc:2454] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafba8ec391942baabc3df18ac19ebb7 in term 1.
I20260809 04:49:56.008085 3225 leader_election.cc:302] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec148d0a5f64c39a0dc36d55284f190, eafba8ec391942baabc3df18ac19ebb7; no voters:
I20260809 04:49:56.008342 3476 raft_consensus.cc:2781] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:49:56.008720 3476 raft_consensus.cc:686] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [term 1 LEADER]: Becoming Leader. State: Replica: eafba8ec391942baabc3df18ac19ebb7, State: Running, Role: LEADER
I20260809 04:49:56.009200 3476 consensus_queue.cc:237] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:56.012797 3499 tablet_replica.cc:439] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7: TxnStatusTablet state changed. Reason: New leader eafba8ec391942baabc3df18ac19ebb7. Latest consensus state: current_term: 1 leader_uuid: "eafba8ec391942baabc3df18ac19ebb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } } }
I20260809 04:49:56.013109 3499 tablet_replica.cc:442] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:49:56.013774 3503 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 04:49:56.013587 2583 catalog_manager.cc:5495] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to eafba8ec391942baabc3df18ac19ebb7 (127.31.93.3). New cstate: current_term: 1 leader_uuid: "eafba8ec391942baabc3df18ac19ebb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } health_report { overall_health: HEALTHY } } }
I20260809 04:49:56.014051 3503 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 04:49:56.072513 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:56.066258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:40111 (local address 127.0.0.1:38968)
0809 04:49:56.066529 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:56.066550 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:56.066724 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:56.067304 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:56.067308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:56.067318 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:49:56.067479 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:56.067486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.068937 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.068941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:56.069989 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:56.070002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.070944 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.070948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:56.071095 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:56.071149 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:49:56.072052 (+ 903us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:49:56.072069 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 04:49:56.072121 (+ 52us) client_negotiation.cc:240] Negotiation successful
0809 04:49:56.072257 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260809 04:49:56.072563 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:56.066363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.31.93.3:40111)
0809 04:49:56.066543 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:56.066550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:56.066894 (+ 344us) server_negotiation.cc:407] Connection header received
0809 04:49:56.067001 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:56.067006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:56.067137 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:56.067281 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:49:56.067603 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.068823 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.070114 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.070805 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.070929 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:56.071315 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:49:56.071964 (+ 649us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:49:56.072101 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:56.072158 (+ 57us) server_negotiation.cc:299] Negotiation successful
0809 04:49:56.072241 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 04:49:56.080534 3106 raft_consensus.cc:1261] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Refusing update from remote peer eafba8ec391942baabc3df18ac19ebb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:49:56.080574 2889 raft_consensus.cc:1261] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 FOLLOWER]: Refusing update from remote peer eafba8ec391942baabc3df18ac19ebb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:49:56.081560 3476 consensus_queue.cc:1035] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:49:56.082262 3499 consensus_queue.cc:1035] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:49:56.093952 3474 tablet_replica.cc:439] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190: TxnStatusTablet state changed. Reason: New leader eafba8ec391942baabc3df18ac19ebb7. Latest consensus state: current_term: 1 leader_uuid: "eafba8ec391942baabc3df18ac19ebb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } } }
I20260809 04:49:56.094429 3474 tablet_replica.cc:442] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:49:56.096467 3472 tablet_replica.cc:439] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a: TxnStatusTablet state changed. Reason: New leader eafba8ec391942baabc3df18ac19ebb7. Latest consensus state: current_term: 1 leader_uuid: "eafba8ec391942baabc3df18ac19ebb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } } }
I20260809 04:49:56.096953 3472 tablet_replica.cc:442] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:49:56.099551 3506 mvcc.cc:205] Tried to move back new op lower bound from 7316484079936229376 to 7316484079657242624. Current Snapshot: MvccSnapshot[applied={T|T < 7316484079936229376}]
I20260809 04:49:56.102447 3472 tablet_replica.cc:439] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eafba8ec391942baabc3df18ac19ebb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } } }
I20260809 04:49:56.102015 3476 tablet_replica.cc:439] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eafba8ec391942baabc3df18ac19ebb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } } }
I20260809 04:49:56.103422 3476 tablet_replica.cc:442] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:49:56.104278 3472 tablet_replica.cc:442] T 4e023dab31014ef3aeb23714b6b0672a P 4c0c20ce539b4721a5d4c2a543cd227a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:49:56.104436 3499 tablet_replica.cc:439] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eafba8ec391942baabc3df18ac19ebb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } } }
I20260809 04:49:56.104810 3499 tablet_replica.cc:442] T 4e023dab31014ef3aeb23714b6b0672a P eafba8ec391942baabc3df18ac19ebb7: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:49:56.107630 3481 consensus_queue.cc:1035] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:49:56.110251 3474 tablet_replica.cc:439] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eafba8ec391942baabc3df18ac19ebb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } } }
I20260809 04:49:56.110599 3474 tablet_replica.cc:442] T 4e023dab31014ef3aeb23714b6b0672a P 5ec148d0a5f64c39a0dc36d55284f190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:49:56.117223 3472 consensus_queue.cc:1035] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:49:56.129804 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:56.121202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36110 (local address 127.31.93.1:35287)
0809 04:49:56.121421 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:56.121428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:56.123224 (+ 1796us) server_negotiation.cc:407] Connection header received
0809 04:49:56.123568 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:56.123572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:56.123691 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:56.123804 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:49:56.124586 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.125358 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.127688 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.128122 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.128215 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:56.128748 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:49:56.129210 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:49:56.129310 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:56.129435 (+ 125us) server_negotiation.cc:299] Negotiation successful
0809 04:49:56.129517 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 04:49:56.149843 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:56.141634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:42011 (local address 127.31.93.3:40111)
0809 04:49:56.141863 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:56.141869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:56.143080 (+ 1211us) server_negotiation.cc:407] Connection header received
0809 04:49:56.143269 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:56.143272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:56.143334 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:56.143401 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:49:56.144381 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.145243 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.147809 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.148414 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.148543 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:56.149082 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:49:56.149185 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:49:56.149255 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:56.149451 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 04:49:56.149514 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 04:49:56.150074 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:56.141793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:40111 (local address 127.31.93.1:42011)
0809 04:49:56.142970 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:56.142990 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:56.143146 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:56.143445 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:56.143450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:56.143464 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:49:56.144142 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:56.144157 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.145380 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.145386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:56.147668 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:56.147686 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.148607 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.148614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:56.148792 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:56.148863 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:49:56.149285 (+ 422us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:49:56.149347 (+ 62us) client_negotiation.cc:769] Sending connection context
0809 04:49:56.149441 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 04:49:56.149576 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"thread_start_us":754,"threads_started":1}
I20260809 04:49:56.195537 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:56.186478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:43621 (local address 127.31.93.1:35287)
0809 04:49:56.186649 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:56.186656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:56.188150 (+ 1494us) server_negotiation.cc:407] Connection header received
0809 04:49:56.188393 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:56.188398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:56.188485 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:56.188554 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:49:56.189417 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.190650 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.193334 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.194086 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.194179 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:56.194921 (+ 742us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:49:56.195020 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:49:56.195086 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:56.195230 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 04:49:56.195289 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 04:49:56.196055 3539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:56.186845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:35287 (local address 127.31.93.3:43621)
0809 04:49:56.188007 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:56.188027 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:56.188263 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:56.188597 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:56.188602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:56.188614 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:49:56.189223 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:56.189239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.190862 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.190868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:56.193175 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:56.193192 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.194444 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.194452 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:56.194690 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:56.194760 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:49:56.195117 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:49:56.195129 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 04:49:56.195216 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 04:49:56.195357 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"thread_start_us":658,"threads_started":1}
I20260809 04:49:56.238803 2583 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:49:56.239126 2583 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:49:56.243310 2583 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c: Sending DeleteTablet for 3 replicas of tablet 1160e45e410b4ab69c80a74d91448b01
I20260809 04:49:56.244653 2843 tablet_service.cc:1514] Processing DeleteTablet for tablet 1160e45e410b4ab69c80a74d91448b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:49:56 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:36102
I20260809 04:49:56.245472 3068 tablet_service.cc:1514] Processing DeleteTablet for tablet 1160e45e410b4ab69c80a74d91448b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:49:56 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:44236
I20260809 04:49:56.246382 3322 tablet_service.cc:1514] Processing DeleteTablet for tablet 1160e45e410b4ab69c80a74d91448b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:49:56 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:38954
I20260809 04:49:56.248631 2583 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:49:56.250231 2583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:49:56.256073 3543 tablet_replica.cc:330] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a: stopping tablet replica
I20260809 04:49:56.256601 3543 raft_consensus.cc:2229] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:49:56.257519 3543 raft_consensus.cc:2258] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:49:56.259555 2843 tablet_service.cc:1467] Processing CreateTablet for tablet 10d824e11f474ecabb772d29b74dedfa (DEFAULT_TABLE table=test-table [id=3c69bb3c8103441aac664d0466099893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:56.260097 2843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10d824e11f474ecabb772d29b74dedfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:56.260645 3066 tablet_service.cc:1467] Processing CreateTablet for tablet 10d824e11f474ecabb772d29b74dedfa (DEFAULT_TABLE table=test-table [id=3c69bb3c8103441aac664d0466099893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:56.260530 3309 tablet_service.cc:1467] Processing CreateTablet for tablet 10d824e11f474ecabb772d29b74dedfa (DEFAULT_TABLE table=test-table [id=3c69bb3c8103441aac664d0466099893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:56.261195 3066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10d824e11f474ecabb772d29b74dedfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:56.261530 3309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10d824e11f474ecabb772d29b74dedfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:56.266947 3543 ts_tablet_manager.cc:1905] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:49:56.269111 3467 tablet_bootstrap.cc:492] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190: Bootstrap starting.
I20260809 04:49:56.269308 3466 tablet_bootstrap.cc:492] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a: Bootstrap starting.
I20260809 04:49:56.271193 3467 tablet_bootstrap.cc:654] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:56.271557 3466 tablet_bootstrap.cc:654] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:56.271974 3468 tablet_bootstrap.cc:492] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7: Bootstrap starting.
I20260809 04:49:56.274168 3468 tablet_bootstrap.cc:654] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:56.276073 3466 tablet_bootstrap.cc:492] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a: No bootstrap required, opened a new log
I20260809 04:49:56.277307 3466 ts_tablet_manager.cc:1397] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260809 04:49:56.277806 3467 tablet_bootstrap.cc:492] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190: No bootstrap required, opened a new log
I20260809 04:49:56.277992 3467 ts_tablet_manager.cc:1397] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260809 04:49:56.278036 3549 tablet_replica.cc:330] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7: stopping tablet replica
I20260809 04:49:56.278038 3466 raft_consensus.cc:348] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:56.278424 3466 raft_consensus.cc:374] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:56.278573 3466 raft_consensus.cc:729] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0c20ce539b4721a5d4c2a543cd227a, State: Initialized, Role: FOLLOWER
I20260809 04:49:56.278874 3468 tablet_bootstrap.cc:492] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7: No bootstrap required, opened a new log
I20260809 04:49:56.279117 3468 ts_tablet_manager.cc:1397] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260809 04:49:56.278880 3466 consensus_queue.cc:260] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:56.279039 3467 raft_consensus.cc:348] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:56.279488 3467 raft_consensus.cc:374] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:56.279670 3467 raft_consensus.cc:729] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec148d0a5f64c39a0dc36d55284f190, State: Initialized, Role: FOLLOWER
I20260809 04:49:56.279999 3466 ts_tablet_manager.cc:1428] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260809 04:49:56.280052 3467 consensus_queue.cc:260] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:56.283155 3467 ts_tablet_manager.cc:1428] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260809 04:49:56.285401 3468 raft_consensus.cc:348] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:56.285817 3468 raft_consensus.cc:374] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:56.286036 3468 raft_consensus.cc:729] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafba8ec391942baabc3df18ac19ebb7, State: Initialized, Role: FOLLOWER
I20260809 04:49:56.286069 3547 tablet_replica.cc:330] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190: stopping tablet replica
I20260809 04:49:56.286486 3547 raft_consensus.cc:2229] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:49:56.286501 3468 consensus_queue.cc:260] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:56.289009 3543 ts_tablet_manager.cc:1918] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:49:56.289326 3543 log.cc:1196] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/1160e45e410b4ab69c80a74d91448b01
I20260809 04:49:56.289386 3468 ts_tablet_manager.cc:1428] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260809 04:49:56.289572 3547 raft_consensus.cc:2258] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:49:56.290527 3549 raft_consensus.cc:2229] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:49:56.290982 3549 raft_consensus.cc:2258] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:49:56.291424 3543 ts_tablet_manager.cc:1939] T 1160e45e410b4ab69c80a74d91448b01 P 4c0c20ce539b4721a5d4c2a543cd227a: Deleting consensus metadata
I20260809 04:49:56.293031 3547 ts_tablet_manager.cc:1905] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:49:56.293157 3549 ts_tablet_manager.cc:1905] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:49:56.294471 2558 catalog_manager.cc:4841] TS 4c0c20ce539b4721a5d4c2a543cd227a (127.31.93.1:35287): tablet 1160e45e410b4ab69c80a74d91448b01 (table test-table [id=1f01630e94404595b02540c17f9e4c14]) successfully deleted
I20260809 04:49:56.302652 3549 ts_tablet_manager.cc:1918] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:49:56.302870 3549 log.cc:1196] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/1160e45e410b4ab69c80a74d91448b01
I20260809 04:49:56.303046 3547 ts_tablet_manager.cc:1918] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:49:56.303265 3547 log.cc:1196] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/1160e45e410b4ab69c80a74d91448b01
I20260809 04:49:56.303445 3549 ts_tablet_manager.cc:1939] T 1160e45e410b4ab69c80a74d91448b01 P eafba8ec391942baabc3df18ac19ebb7: Deleting consensus metadata
I20260809 04:49:56.305189 3547 ts_tablet_manager.cc:1939] T 1160e45e410b4ab69c80a74d91448b01 P 5ec148d0a5f64c39a0dc36d55284f190: Deleting consensus metadata
I20260809 04:49:56.305249 2565 catalog_manager.cc:4841] TS eafba8ec391942baabc3df18ac19ebb7 (127.31.93.3:40111): tablet 1160e45e410b4ab69c80a74d91448b01 (table test-table [id=1f01630e94404595b02540c17f9e4c14]) successfully deleted
I20260809 04:49:56.307082 2557 catalog_manager.cc:4841] TS 5ec148d0a5f64c39a0dc36d55284f190 (127.31.93.2:37635): tablet 1160e45e410b4ab69c80a74d91448b01 (table test-table [id=1f01630e94404595b02540c17f9e4c14]) successfully deleted
I20260809 04:49:56.345052 3476 raft_consensus.cc:482] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:49:56.345320 3476 raft_consensus.cc:504] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:56.346469 3476 leader_election.cc:288] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c0c20ce539b4721a5d4c2a543cd227a (127.31.93.1:35287), 5ec148d0a5f64c39a0dc36d55284f190 (127.31.93.2:37635)
I20260809 04:49:56.347288 3106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d824e11f474ecabb772d29b74dedfa" candidate_uuid: "eafba8ec391942baabc3df18ac19ebb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec148d0a5f64c39a0dc36d55284f190" is_pre_election: true
I20260809 04:49:56.347317 2888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d824e11f474ecabb772d29b74dedfa" candidate_uuid: "eafba8ec391942baabc3df18ac19ebb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" is_pre_election: true
I20260809 04:49:56.347579 2888 raft_consensus.cc:2454] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eafba8ec391942baabc3df18ac19ebb7 in term 0.
I20260809 04:49:56.347615 3106 raft_consensus.cc:2454] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eafba8ec391942baabc3df18ac19ebb7 in term 0.
I20260809 04:49:56.348081 3226 leader_election.cc:302] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c0c20ce539b4721a5d4c2a543cd227a, eafba8ec391942baabc3df18ac19ebb7; no voters:
I20260809 04:49:56.348510 3476 raft_consensus.cc:2781] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:49:56.348661 3476 raft_consensus.cc:482] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:49:56.348778 3476 raft_consensus.cc:3037] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:56.352200 3476 raft_consensus.cc:504] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:56.353240 2888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d824e11f474ecabb772d29b74dedfa" candidate_uuid: "eafba8ec391942baabc3df18ac19ebb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c0c20ce539b4721a5d4c2a543cd227a"
I20260809 04:49:56.353448 2888 raft_consensus.cc:3037] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:56.354904 3106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d824e11f474ecabb772d29b74dedfa" candidate_uuid: "eafba8ec391942baabc3df18ac19ebb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec148d0a5f64c39a0dc36d55284f190"
I20260809 04:49:56.355119 3106 raft_consensus.cc:3037] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:56.357270 2888 raft_consensus.cc:2454] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafba8ec391942baabc3df18ac19ebb7 in term 1.
I20260809 04:49:56.357723 3226 leader_election.cc:302] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c0c20ce539b4721a5d4c2a543cd227a, eafba8ec391942baabc3df18ac19ebb7; no voters:
I20260809 04:49:56.358065 3106 raft_consensus.cc:2454] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafba8ec391942baabc3df18ac19ebb7 in term 1.
I20260809 04:49:56.358073 3521 raft_consensus.cc:2781] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:49:56.358229 3521 raft_consensus.cc:686] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [term 1 LEADER]: Becoming Leader. State: Replica: eafba8ec391942baabc3df18ac19ebb7, State: Running, Role: LEADER
I20260809 04:49:56.358621 3521 consensus_queue.cc:237] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:56.358712 3476 leader_election.cc:288] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [CANDIDATE]: Term 1 election: Requested vote from peers 4c0c20ce539b4721a5d4c2a543cd227a (127.31.93.1:35287), 5ec148d0a5f64c39a0dc36d55284f190 (127.31.93.2:37635)
I20260809 04:49:56.370597 2586 catalog_manager.cc:5495] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to eafba8ec391942baabc3df18ac19ebb7 (127.31.93.3). New cstate: current_term: 1 leader_uuid: "eafba8ec391942baabc3df18ac19ebb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } health_report { overall_health: HEALTHY } } }
I20260809 04:49:56.411298 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:56.403288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38984 (local address 127.31.93.3:40111)
0809 04:49:56.403442 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:56.403447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:56.403465 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:49:56.403522 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:56.403527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:56.403610 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:56.403696 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:49:56.404384 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.405769 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.407969 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.409062 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.409188 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:56.409789 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:49:56.410028 (+ 239us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:49:56.410118 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:56.410742 (+ 624us) server_negotiation.cc:299] Negotiation successful
0809 04:49:56.410817 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:49:56.430975 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:56.422954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:59251 (local address 127.31.93.3:40111)
0809 04:49:56.423124 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:56.423130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:56.423738 (+ 608us) server_negotiation.cc:407] Connection header received
0809 04:49:56.423891 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:56.423900 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:56.423976 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:56.424074 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:49:56.424826 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.426179 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.428585 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.429349 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.429472 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:56.429973 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:49:56.430092 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:49:56.430183 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:56.430376 (+ 193us) server_negotiation.cc:299] Negotiation successful
0809 04:49:56.430476 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":57}
I20260809 04:49:56.431021 3539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:56.423422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:40111 (local address 127.31.93.3:59251)
0809 04:49:56.423628 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 04:49:56.423647 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:49:56.423764 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:49:56.424248 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:49:56.424252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:49:56.424268 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:49:56.424659 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:56.424672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.426397 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.426406 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:56.428430 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:49:56.428445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:49:56.429505 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:49:56.429512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:49:56.429711 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:56.429758 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:49:56.430200 (+ 442us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:49:56.430211 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 04:49:56.430315 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 04:49:56.430512 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":38}
I20260809 04:49:56.441618 3106 raft_consensus.cc:1261] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Refusing update from remote peer eafba8ec391942baabc3df18ac19ebb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:49:56.442195 3521 consensus_queue.cc:1035] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:49:56.443512 2888 raft_consensus.cc:1261] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 FOLLOWER]: Refusing update from remote peer eafba8ec391942baabc3df18ac19ebb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:49:56.444196 3499 consensus_queue.cc:1035] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:49:56.459462 3510 mvcc.cc:205] Tried to move back new op lower bound from 7316484081419894784 to 7316484081089069056. Current Snapshot: MvccSnapshot[applied={T|T < 7316484081419894784}]
I20260809 04:49:56.466806 3512 mvcc.cc:205] Tried to move back new op lower bound from 7316484081419894784 to 7316484081089069056. Current Snapshot: MvccSnapshot[applied={T|T < 7316484081419894784}]
W20260809 04:49:56.485358 3506 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 04:49:58.089546 2586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:49:58.089810 2586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:49:58.093991 2586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c: Sending DeleteTablet for 3 replicas of tablet 10d824e11f474ecabb772d29b74dedfa
I20260809 04:49:58.094868 3068 tablet_service.cc:1514] Processing DeleteTablet for tablet 10d824e11f474ecabb772d29b74dedfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:49:58 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:44236
I20260809 04:49:58.094811 2843 tablet_service.cc:1514] Processing DeleteTablet for tablet 10d824e11f474ecabb772d29b74dedfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:49:58 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:36102
I20260809 04:49:58.094901 3322 tablet_service.cc:1514] Processing DeleteTablet for tablet 10d824e11f474ecabb772d29b74dedfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:49:58 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:38954
I20260809 04:49:58.107548 3598 tablet_replica.cc:330] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190: stopping tablet replica
I20260809 04:49:58.107999 3600 tablet_replica.cc:330] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7: stopping tablet replica
I20260809 04:49:58.108268 3600 raft_consensus.cc:2229] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:49:58.108897 3600 raft_consensus.cc:2258] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:49:58.108850 3598 raft_consensus.cc:2229] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:49:58.109174 3598 raft_consensus.cc:2258] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:49:58.109869 3600 ts_tablet_manager.cc:1905] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:49:58.110802 3598 ts_tablet_manager.cc:1905] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:49:58.111042 3597 tablet_replica.cc:330] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a: stopping tablet replica
I20260809 04:49:58.111380 3597 raft_consensus.cc:2229] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:49:58.111704 3597 raft_consensus.cc:2258] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:49:58.112699 3597 ts_tablet_manager.cc:1905] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:49:58.117292 3600 ts_tablet_manager.cc:1918] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 04:49:58.117624 3600 log.cc:1196] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/10d824e11f474ecabb772d29b74dedfa
I20260809 04:49:58.118276 3598 ts_tablet_manager.cc:1918] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 04:49:58.118330 3600 ts_tablet_manager.cc:1939] T 10d824e11f474ecabb772d29b74dedfa P eafba8ec391942baabc3df18ac19ebb7: Deleting consensus metadata
I20260809 04:49:58.118541 3598 log.cc:1196] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/10d824e11f474ecabb772d29b74dedfa
I20260809 04:49:58.119184 3598 ts_tablet_manager.cc:1939] T 10d824e11f474ecabb772d29b74dedfa P 5ec148d0a5f64c39a0dc36d55284f190: Deleting consensus metadata
I20260809 04:49:58.119505 2565 catalog_manager.cc:4841] TS eafba8ec391942baabc3df18ac19ebb7 (127.31.93.3:40111): tablet 10d824e11f474ecabb772d29b74dedfa (table test-table [id=3c69bb3c8103441aac664d0466099893]) successfully deleted
I20260809 04:49:58.120350 2557 catalog_manager.cc:4841] TS 5ec148d0a5f64c39a0dc36d55284f190 (127.31.93.2:37635): tablet 10d824e11f474ecabb772d29b74dedfa (table test-table [id=3c69bb3c8103441aac664d0466099893]) successfully deleted
I20260809 04:49:58.120426 3597 ts_tablet_manager.cc:1918] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 04:49:58.120615 3597 log.cc:1196] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/10d824e11f474ecabb772d29b74dedfa
I20260809 04:49:58.121129 3597 ts_tablet_manager.cc:1939] T 10d824e11f474ecabb772d29b74dedfa P 4c0c20ce539b4721a5d4c2a543cd227a: Deleting consensus metadata
I20260809 04:49:58.122008 2558 catalog_manager.cc:4841] TS 4c0c20ce539b4721a5d4c2a543cd227a (127.31.93.1:35287): tablet 10d824e11f474ecabb772d29b74dedfa (table test-table [id=3c69bb3c8103441aac664d0466099893]) successfully deleted
Aug 09 04:49:58 dist-test-slave-4tdj krb5kdc[2511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 04:49:58 dist-test-slave-4tdj krb5kdc[2511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:49:58.156243 3610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:58.141598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.31.93.62:42363)
0809 04:49:58.142353 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:58.142357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:58.142413 (+ 56us) server_negotiation.cc:407] Connection header received
0809 04:49:58.142470 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:58.142474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:58.142536 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:58.142630 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:49:58.144123 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:58.145103 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:58.146585 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:58.147136 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:58.147233 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:58.151165 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:49:58.151172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:49:58.151175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:49:58.151210 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:49:58.153976 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:58.154608 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:58.154611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:58.154614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:58.154735 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:49:58.154991 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:49:58.154994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:49:58.154996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:49:58.155265 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:49:58.155436 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:58.155650 (+ 214us) server_negotiation.cc:299] Negotiation successful
0809 04:49:58.155821 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":365,"threads_started":1}
I20260809 04:49:58.159786 2586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:49:58.160418 2586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:49:58.167155 2843 tablet_service.cc:1467] Processing CreateTablet for tablet d96d1a1d8abc49869983a62b39288e19 (DEFAULT_TABLE table=test-table [id=1da98fc8ad524feabe8ade57eedddd51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:58.167636 2843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96d1a1d8abc49869983a62b39288e19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:58.167598 3322 tablet_service.cc:1467] Processing CreateTablet for tablet d96d1a1d8abc49869983a62b39288e19 (DEFAULT_TABLE table=test-table [id=1da98fc8ad524feabe8ade57eedddd51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:58.167614 3068 tablet_service.cc:1467] Processing CreateTablet for tablet d96d1a1d8abc49869983a62b39288e19 (DEFAULT_TABLE table=test-table [id=1da98fc8ad524feabe8ade57eedddd51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:58.167979 3322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96d1a1d8abc49869983a62b39288e19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:58.168023 3068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d96d1a1d8abc49869983a62b39288e19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:58.174192 3617 tablet_bootstrap.cc:492] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7: Bootstrap starting.
I20260809 04:49:58.175595 3618 tablet_bootstrap.cc:492] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a: Bootstrap starting.
I20260809 04:49:58.176698 3617 tablet_bootstrap.cc:654] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:58.178012 3618 tablet_bootstrap.cc:654] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:58.179158 3619 tablet_bootstrap.cc:492] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190: Bootstrap starting.
I20260809 04:49:58.180053 3617 tablet_bootstrap.cc:492] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7: No bootstrap required, opened a new log
I20260809 04:49:58.180233 3617 ts_tablet_manager.cc:1397] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:49:58.181466 3617 raft_consensus.cc:348] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.181718 3619 tablet_bootstrap.cc:654] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:58.181828 3617 raft_consensus.cc:374] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:58.182021 3617 raft_consensus.cc:729] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafba8ec391942baabc3df18ac19ebb7, State: Initialized, Role: FOLLOWER
I20260809 04:49:58.182317 3617 consensus_queue.cc:260] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.182796 3618 tablet_bootstrap.cc:492] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a: No bootstrap required, opened a new log
I20260809 04:49:58.183111 3618 ts_tablet_manager.cc:1397] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260809 04:49:58.183143 3617 ts_tablet_manager.cc:1428] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260809 04:49:58.184293 3618 raft_consensus.cc:348] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.184595 3618 raft_consensus.cc:374] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:58.184769 3618 raft_consensus.cc:729] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0c20ce539b4721a5d4c2a543cd227a, State: Initialized, Role: FOLLOWER
I20260809 04:49:58.185101 3618 consensus_queue.cc:260] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.185344 3619 tablet_bootstrap.cc:492] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190: No bootstrap required, opened a new log
I20260809 04:49:58.185487 3619 ts_tablet_manager.cc:1397] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260809 04:49:58.186604 3619 raft_consensus.cc:348] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.186952 3619 raft_consensus.cc:374] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:58.187101 3619 raft_consensus.cc:729] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec148d0a5f64c39a0dc36d55284f190, State: Initialized, Role: FOLLOWER
I20260809 04:49:58.187341 3618 ts_tablet_manager.cc:1428] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260809 04:49:58.187397 3619 consensus_queue.cc:260] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.189474 3619 ts_tablet_manager.cc:1428] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260809 04:49:58.356734 3633 raft_consensus.cc:482] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:49:58.357172 3633 raft_consensus.cc:504] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.358194 3633 leader_election.cc:288] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ec148d0a5f64c39a0dc36d55284f190 (127.31.93.2:37635), eafba8ec391942baabc3df18ac19ebb7 (127.31.93.3:40111)
I20260809 04:49:58.358769 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96d1a1d8abc49869983a62b39288e19" candidate_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafba8ec391942baabc3df18ac19ebb7" is_pre_election: true
I20260809 04:49:58.358871 3106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96d1a1d8abc49869983a62b39288e19" candidate_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec148d0a5f64c39a0dc36d55284f190" is_pre_election: true
I20260809 04:49:58.359043 3360 raft_consensus.cc:2454] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c0c20ce539b4721a5d4c2a543cd227a in term 0.
I20260809 04:49:58.359206 3106 raft_consensus.cc:2454] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c0c20ce539b4721a5d4c2a543cd227a in term 0.
I20260809 04:49:58.359612 2757 leader_election.cc:302] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c0c20ce539b4721a5d4c2a543cd227a, eafba8ec391942baabc3df18ac19ebb7; no voters:
I20260809 04:49:58.359912 3633 raft_consensus.cc:2781] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:49:58.360055 3633 raft_consensus.cc:482] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:49:58.360241 3633 raft_consensus.cc:3037] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:58.363348 3633 raft_consensus.cc:504] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.363852 3633 leader_election.cc:288] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [CANDIDATE]: Term 1 election: Requested vote from peers 5ec148d0a5f64c39a0dc36d55284f190 (127.31.93.2:37635), eafba8ec391942baabc3df18ac19ebb7 (127.31.93.3:40111)
I20260809 04:49:58.364252 3106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96d1a1d8abc49869983a62b39288e19" candidate_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec148d0a5f64c39a0dc36d55284f190"
I20260809 04:49:58.364444 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d96d1a1d8abc49869983a62b39288e19" candidate_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eafba8ec391942baabc3df18ac19ebb7"
I20260809 04:49:58.364502 3106 raft_consensus.cc:3037] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:58.364662 3360 raft_consensus.cc:3037] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:58.366905 3106 raft_consensus.cc:2454] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c0c20ce539b4721a5d4c2a543cd227a in term 1.
I20260809 04:49:58.367295 2753 leader_election.cc:302] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c0c20ce539b4721a5d4c2a543cd227a, 5ec148d0a5f64c39a0dc36d55284f190; no voters:
I20260809 04:49:58.367502 3360 raft_consensus.cc:2454] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c0c20ce539b4721a5d4c2a543cd227a in term 1.
I20260809 04:49:58.367645 3633 raft_consensus.cc:2781] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:49:58.368269 3633 raft_consensus.cc:686] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 LEADER]: Becoming Leader. State: Replica: 4c0c20ce539b4721a5d4c2a543cd227a, State: Running, Role: LEADER
I20260809 04:49:58.368774 3633 consensus_queue.cc:237] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.371313 2586 catalog_manager.cc:5495] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c0c20ce539b4721a5d4c2a543cd227a (127.31.93.1). New cstate: current_term: 1 leader_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } health_report { overall_health: UNKNOWN } } }
I20260809 04:49:58.427811 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:58.421589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.31.93.1:35287)
0809 04:49:58.422399 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:58.422404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:58.422427 (+ 23us) server_negotiation.cc:407] Connection header received
0809 04:49:58.422475 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:58.422478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:58.422538 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:58.422668 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:49:58.423140 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:58.424255 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:58.425691 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:58.426147 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:58.426332 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:58.426794 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:49:58.426929 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:49:58.427008 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:58.427159 (+ 151us) server_negotiation.cc:299] Negotiation successful
0809 04:49:58.427237 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":466,"threads_started":1}
I20260809 04:49:58.436755 3360 raft_consensus.cc:1261] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7 [term 1 FOLLOWER]: Refusing update from remote peer 4c0c20ce539b4721a5d4c2a543cd227a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:49:58.436856 3106 raft_consensus.cc:1261] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Refusing update from remote peer 4c0c20ce539b4721a5d4c2a543cd227a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:49:58.437469 3637 consensus_queue.cc:1035] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:49:58.437983 3633 consensus_queue.cc:1035] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:49:58.482887 2586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:49:58.483095 2586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:49:58.485967 2586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c: Sending DeleteTablet for 3 replicas of tablet d96d1a1d8abc49869983a62b39288e19
I20260809 04:49:58.486681 2843 tablet_service.cc:1514] Processing DeleteTablet for tablet d96d1a1d8abc49869983a62b39288e19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:49:58 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:36102
I20260809 04:49:58.487011 3068 tablet_service.cc:1514] Processing DeleteTablet for tablet d96d1a1d8abc49869983a62b39288e19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:49:58 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:44236
I20260809 04:49:58.487066 3597 tablet_replica.cc:330] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a: stopping tablet replica
I20260809 04:49:58.487053 3322 tablet_service.cc:1514] Processing DeleteTablet for tablet d96d1a1d8abc49869983a62b39288e19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:49:58 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:38954
I20260809 04:49:58.487315 3598 tablet_replica.cc:330] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190: stopping tablet replica
I20260809 04:49:58.487347 3600 tablet_replica.cc:330] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7: stopping tablet replica
I20260809 04:49:58.487411 3597 raft_consensus.cc:2229] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:49:58.487571 3600 raft_consensus.cc:2229] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:49:58.487612 3598 raft_consensus.cc:2229] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:49:58.487752 3600 raft_consensus.cc:2258] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:49:58.487807 3597 raft_consensus.cc:2258] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:49:58.487838 3598 raft_consensus.cc:2258] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:49:58.488605 2586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 04:49:58.488983 3600 ts_tablet_manager.cc:1905] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:49:58.488983 3598 ts_tablet_manager.cc:1905] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 04:49:58.489164 2586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:49:58.490955 3597 ts_tablet_manager.cc:1905] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:49:58.496289 3600 ts_tablet_manager.cc:1918] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:49:58.496447 3600 log.cc:1196] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/d96d1a1d8abc49869983a62b39288e19
I20260809 04:49:58.496690 3598 ts_tablet_manager.cc:1918] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:49:58.496853 3598 log.cc:1196] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/d96d1a1d8abc49869983a62b39288e19
I20260809 04:49:58.497243 3598 ts_tablet_manager.cc:1939] T d96d1a1d8abc49869983a62b39288e19 P 5ec148d0a5f64c39a0dc36d55284f190: Deleting consensus metadata
I20260809 04:49:58.497244 3600 ts_tablet_manager.cc:1939] T d96d1a1d8abc49869983a62b39288e19 P eafba8ec391942baabc3df18ac19ebb7: Deleting consensus metadata
I20260809 04:49:58.498518 3597 ts_tablet_manager.cc:1918] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:49:58.498468 2557 catalog_manager.cc:4841] TS 5ec148d0a5f64c39a0dc36d55284f190 (127.31.93.2:37635): tablet d96d1a1d8abc49869983a62b39288e19 (table test-table [id=1da98fc8ad524feabe8ade57eedddd51]) successfully deleted
I20260809 04:49:58.498785 3597 log.cc:1196] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/d96d1a1d8abc49869983a62b39288e19
I20260809 04:49:58.498725 3068 tablet_service.cc:1467] Processing CreateTablet for tablet 76ade5292ec8431b9e7ec125a12809e2 (DEFAULT_TABLE table=test-table [id=d6defa2bf7aa4c22872b792107636d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:58.498965 2565 catalog_manager.cc:4841] TS eafba8ec391942baabc3df18ac19ebb7 (127.31.93.3:40111): tablet d96d1a1d8abc49869983a62b39288e19 (table test-table [id=1da98fc8ad524feabe8ade57eedddd51]) successfully deleted
I20260809 04:49:58.499034 3322 tablet_service.cc:1467] Processing CreateTablet for tablet 76ade5292ec8431b9e7ec125a12809e2 (DEFAULT_TABLE table=test-table [id=d6defa2bf7aa4c22872b792107636d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:58.499212 3068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76ade5292ec8431b9e7ec125a12809e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:58.499408 3322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76ade5292ec8431b9e7ec125a12809e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:58.499755 3597 ts_tablet_manager.cc:1939] T d96d1a1d8abc49869983a62b39288e19 P 4c0c20ce539b4721a5d4c2a543cd227a: Deleting consensus metadata
I20260809 04:49:58.502681 2558 catalog_manager.cc:4841] TS 4c0c20ce539b4721a5d4c2a543cd227a (127.31.93.1:35287): tablet d96d1a1d8abc49869983a62b39288e19 (table test-table [id=1da98fc8ad524feabe8ade57eedddd51]) successfully deleted
I20260809 04:49:58.503041 2843 tablet_service.cc:1467] Processing CreateTablet for tablet 76ade5292ec8431b9e7ec125a12809e2 (DEFAULT_TABLE table=test-table [id=d6defa2bf7aa4c22872b792107636d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:49:58.503497 2843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76ade5292ec8431b9e7ec125a12809e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:49:58.506464 3619 tablet_bootstrap.cc:492] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190: Bootstrap starting.
I20260809 04:49:58.506888 3617 tablet_bootstrap.cc:492] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7: Bootstrap starting.
I20260809 04:49:58.508374 3617 tablet_bootstrap.cc:654] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:58.508937 3619 tablet_bootstrap.cc:654] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:58.510494 3618 tablet_bootstrap.cc:492] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a: Bootstrap starting.
I20260809 04:49:58.511260 3617 tablet_bootstrap.cc:492] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7: No bootstrap required, opened a new log
I20260809 04:49:58.511473 3617 ts_tablet_manager.cc:1397] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260809 04:49:58.512640 3619 tablet_bootstrap.cc:492] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190: No bootstrap required, opened a new log
I20260809 04:49:58.512650 3618 tablet_bootstrap.cc:654] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a: Neither blocks nor log segments found. Creating new log.
I20260809 04:49:58.512903 3619 ts_tablet_manager.cc:1397] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 04:49:58.512815 3617 raft_consensus.cc:348] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.513118 3617 raft_consensus.cc:374] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:58.513264 3617 raft_consensus.cc:729] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafba8ec391942baabc3df18ac19ebb7, State: Initialized, Role: FOLLOWER
I20260809 04:49:58.513558 3617 consensus_queue.cc:260] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.513715 3619 raft_consensus.cc:348] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.514019 3619 raft_consensus.cc:374] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:58.514186 3619 raft_consensus.cc:729] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec148d0a5f64c39a0dc36d55284f190, State: Initialized, Role: FOLLOWER
I20260809 04:49:58.514585 3619 consensus_queue.cc:260] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.514864 3617 ts_tablet_manager.cc:1428] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260809 04:49:58.516335 3618 tablet_bootstrap.cc:492] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a: No bootstrap required, opened a new log
I20260809 04:49:58.516515 3618 ts_tablet_manager.cc:1397] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260809 04:49:58.515703 3619 ts_tablet_manager.cc:1428] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 04:49:58.517354 3618 raft_consensus.cc:348] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.517659 3618 raft_consensus.cc:374] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:49:58.517755 3618 raft_consensus.cc:729] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0c20ce539b4721a5d4c2a543cd227a, State: Initialized, Role: FOLLOWER
I20260809 04:49:58.518000 3618 consensus_queue.cc:260] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.518669 3618 ts_tablet_manager.cc:1428] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260809 04:49:58.526750 3591 raft_consensus.cc:482] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:49:58.526889 3591 raft_consensus.cc:504] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.527784 3591 leader_election.cc:288] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c0c20ce539b4721a5d4c2a543cd227a (127.31.93.1:35287), 5ec148d0a5f64c39a0dc36d55284f190 (127.31.93.2:37635)
I20260809 04:49:58.528257 3105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76ade5292ec8431b9e7ec125a12809e2" candidate_uuid: "eafba8ec391942baabc3df18ac19ebb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec148d0a5f64c39a0dc36d55284f190" is_pre_election: true
I20260809 04:49:58.528505 3105 raft_consensus.cc:2454] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eafba8ec391942baabc3df18ac19ebb7 in term 0.
I20260809 04:49:58.528327 2888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76ade5292ec8431b9e7ec125a12809e2" candidate_uuid: "eafba8ec391942baabc3df18ac19ebb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" is_pre_election: true
I20260809 04:49:58.528756 2888 raft_consensus.cc:2454] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eafba8ec391942baabc3df18ac19ebb7 in term 0.
I20260809 04:49:58.528939 3225 leader_election.cc:302] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec148d0a5f64c39a0dc36d55284f190, eafba8ec391942baabc3df18ac19ebb7; no voters:
I20260809 04:49:58.529611 3591 raft_consensus.cc:2781] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:49:58.529718 3591 raft_consensus.cc:482] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:49:58.529807 3591 raft_consensus.cc:3037] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:58.532133 3591 raft_consensus.cc:504] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.532639 3591 leader_election.cc:288] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [CANDIDATE]: Term 1 election: Requested vote from peers 4c0c20ce539b4721a5d4c2a543cd227a (127.31.93.1:35287), 5ec148d0a5f64c39a0dc36d55284f190 (127.31.93.2:37635)
I20260809 04:49:58.533107 3105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76ade5292ec8431b9e7ec125a12809e2" candidate_uuid: "eafba8ec391942baabc3df18ac19ebb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec148d0a5f64c39a0dc36d55284f190"
I20260809 04:49:58.533128 2888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76ade5292ec8431b9e7ec125a12809e2" candidate_uuid: "eafba8ec391942baabc3df18ac19ebb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c0c20ce539b4721a5d4c2a543cd227a"
I20260809 04:49:58.533267 3105 raft_consensus.cc:3037] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:58.533375 2888 raft_consensus.cc:3037] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:49:58.535878 3105 raft_consensus.cc:2454] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafba8ec391942baabc3df18ac19ebb7 in term 1.
I20260809 04:49:58.536319 3225 leader_election.cc:302] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec148d0a5f64c39a0dc36d55284f190, eafba8ec391942baabc3df18ac19ebb7; no voters:
I20260809 04:49:58.536525 2888 raft_consensus.cc:2454] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafba8ec391942baabc3df18ac19ebb7 in term 1.
I20260809 04:49:58.536624 3591 raft_consensus.cc:2781] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:49:58.537014 3591 raft_consensus.cc:686] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [term 1 LEADER]: Becoming Leader. State: Replica: eafba8ec391942baabc3df18ac19ebb7, State: Running, Role: LEADER
I20260809 04:49:58.537338 3591 consensus_queue.cc:237] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } }
I20260809 04:49:58.540326 2585 catalog_manager.cc:5495] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to eafba8ec391942baabc3df18ac19ebb7 (127.31.93.3). New cstate: current_term: 1 leader_uuid: "eafba8ec391942baabc3df18ac19ebb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eafba8ec391942baabc3df18ac19ebb7" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40111 } health_report { overall_health: HEALTHY } } }
I20260809 04:49:58.567999 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:49:58.561674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35938 (local address 127.31.93.3:40111)
0809 04:49:58.562202 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0809 04:49:58.562208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:49:58.562284 (+ 76us) server_negotiation.cc:407] Connection header received
0809 04:49:58.562483 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:49:58.562496 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:49:58.562562 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:49:58.562644 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:49:58.563249 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:58.564386 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:58.565923 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:49:58.566422 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:49:58.566540 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:49:58.567107 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:49:58.567222 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:49:58.567296 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 04:49:58.567436 (+ 140us) server_negotiation.cc:299] Negotiation successful
0809 04:49:58.567517 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":281,"threads_started":1}
I20260809 04:49:58.576937 2888 raft_consensus.cc:1261] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 FOLLOWER]: Refusing update from remote peer eafba8ec391942baabc3df18ac19ebb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:49:58.576938 3105 raft_consensus.cc:1261] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Refusing update from remote peer eafba8ec391942baabc3df18ac19ebb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:49:58.577468 3572 consensus_queue.cc:1035] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c0c20ce539b4721a5d4c2a543cd227a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 35287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:49:58.577832 3566 consensus_queue.cc:1035] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec148d0a5f64c39a0dc36d55284f190" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 04:49:58.596148 3561 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 04:50:00.176219 2586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:00.176578 2586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:00.180815 2586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b1bf0d5a1e4431499b0d7e0e2a8364c: Sending DeleteTablet for 3 replicas of tablet 76ade5292ec8431b9e7ec125a12809e2
I20260809 04:50:00.181600 2843 tablet_service.cc:1514] Processing DeleteTablet for tablet 76ade5292ec8431b9e7ec125a12809e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:00 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:36102
I20260809 04:50:00.181596 3068 tablet_service.cc:1514] Processing DeleteTablet for tablet 76ade5292ec8431b9e7ec125a12809e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:00 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:44236
I20260809 04:50:00.182019 3322 tablet_service.cc:1514] Processing DeleteTablet for tablet 76ade5292ec8431b9e7ec125a12809e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:00 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:38954
I20260809 04:50:00.182502 3695 tablet_replica.cc:330] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190: stopping tablet replica
I20260809 04:50:00.182863 3695 raft_consensus.cc:2229] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:00.183234 3695 raft_consensus.cc:2258] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:00.183952 3697 tablet_replica.cc:330] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a: stopping tablet replica
I20260809 04:50:00.182533 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 2724
I20260809 04:50:00.184268 3697 raft_consensus.cc:2229] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:00.184635 3697 raft_consensus.cc:2258] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:00.184619 3695 ts_tablet_manager.cc:1905] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:00.185393 3699 tablet_replica.cc:330] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7: stopping tablet replica
I20260809 04:50:00.185667 3699 raft_consensus.cc:2229] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:50:00.186075 3699 raft_consensus.cc:2258] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:00.186892 3697 ts_tablet_manager.cc:1905] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:00.187623 3699 ts_tablet_manager.cc:1905] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:00.192755 3695 ts_tablet_manager.cc:1918] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 04:50:00.192984 3695 log.cc:1196] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/76ade5292ec8431b9e7ec125a12809e2
I20260809 04:50:00.193727 3695 ts_tablet_manager.cc:1939] T 76ade5292ec8431b9e7ec125a12809e2 P 5ec148d0a5f64c39a0dc36d55284f190: Deleting consensus metadata
I20260809 04:50:00.194633 3697 ts_tablet_manager.cc:1918] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 04:50:00.194808 3697 log.cc:1196] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/76ade5292ec8431b9e7ec125a12809e2
I20260809 04:50:00.194904 2557 catalog_manager.cc:4841] TS 5ec148d0a5f64c39a0dc36d55284f190 (127.31.93.2:37635): tablet 76ade5292ec8431b9e7ec125a12809e2 (table test-table [id=d6defa2bf7aa4c22872b792107636d33]) successfully deleted
I20260809 04:50:00.195420 3697 ts_tablet_manager.cc:1939] T 76ade5292ec8431b9e7ec125a12809e2 P 4c0c20ce539b4721a5d4c2a543cd227a: Deleting consensus metadata
I20260809 04:50:00.195500 3699 ts_tablet_manager.cc:1918] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 04:50:00.195716 3699 log.cc:1196] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/76ade5292ec8431b9e7ec125a12809e2
I20260809 04:50:00.196331 3699 ts_tablet_manager.cc:1939] T 76ade5292ec8431b9e7ec125a12809e2 P eafba8ec391942baabc3df18ac19ebb7: Deleting consensus metadata
I20260809 04:50:00.196470 2558 catalog_manager.cc:4841] TS 4c0c20ce539b4721a5d4c2a543cd227a (127.31.93.1:35287): tablet 76ade5292ec8431b9e7ec125a12809e2 (table test-table [id=d6defa2bf7aa4c22872b792107636d33]) successfully deleted
I20260809 04:50:00.197288 2565 catalog_manager.cc:4841] TS eafba8ec391942baabc3df18ac19ebb7 (127.31.93.3:40111): tablet 76ade5292ec8431b9e7ec125a12809e2 (table test-table [id=d6defa2bf7aa4c22872b792107636d33]) successfully deleted
Aug 09 04:50:00 dist-test-slave-4tdj krb5kdc[2511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:50:00.212543 3702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:00.185695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42650 (local address 127.31.93.1:35287)
0809 04:50:00.196796 (+ 11101us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:00.196800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:00.196817 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:50:00.196862 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:00.196865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:00.196924 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:00.197053 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:00.198416 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:00.199353 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:00.200507 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:00.200914 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:00.201004 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:00.205870 (+ 4866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:00.205884 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:00.205902 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:00.205932 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:00.209663 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:00.210286 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:00.210353 (+ 67us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:00.210357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:00.210546 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:00.210966 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:00.210971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:00.210975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:00.211418 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:00.211738 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:00.211853 (+ 115us) server_negotiation.cc:299] Negotiation successful
0809 04:50:00.212060 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10994,"thread_start_us":10805,"threads_started":1}
W20260809 04:50:00.213388 2951 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42650
W20260809 04:50:00.214114 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 2724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:50:00.214473 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 2724
I20260809 04:50:00.246510 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 2964
Aug 09 04:50:00 dist-test-slave-4tdj krb5kdc[2511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:50:00.263351 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:00.247241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.31.93.2:37635)
0809 04:50:00.247954 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:00.247958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:00.247975 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:50:00.248031 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:00.248035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:00.248106 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:00.248196 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:00.249323 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:00.250098 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:00.251278 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:00.251671 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:00.251760 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:00.258113 (+ 6353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:00.258129 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:00.258144 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:00.258182 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:00.261054 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:00.261622 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:00.261629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:00.261631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:00.261792 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:00.262099 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:00.262102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:00.262104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:00.262467 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:00.262683 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:00.262916 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 04:50:00.263074 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":457,"threads_started":1}
W20260809 04:50:00.264010 3186 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56776
W20260809 04:50:00.264572 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 2964: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:50:00.264719 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 2964
I20260809 04:50:00.292033 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 3204
Aug 09 04:50:00 dist-test-slave-4tdj krb5kdc[2511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786250998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:50:00.307619 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:00.292828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35940 (local address 127.31.93.3:40111)
0809 04:50:00.293650 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:00.293656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:00.293672 (+ 16us) server_negotiation.cc:407] Connection header received
0809 04:50:00.293727 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:00.293730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:00.293788 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:00.293883 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:00.295312 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:00.296307 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:00.297537 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:00.297956 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:00.298064 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:00.302107 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:00.302123 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:00.302140 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:00.302189 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:00.304894 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:00.305571 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:00.305578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:00.305581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:00.305727 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:00.306076 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:00.306080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:00.306082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:00.306502 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:00.306817 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:00.307072 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 04:50:00.307304 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":434,"threads_started":1}
W20260809 04:50:00.308308 3430 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35940
W20260809 04:50:00.308893 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 3204: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:50:00.309052 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 3204
I20260809 04:50:00.339550 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 2533
I20260809 04:50:00.340693 2666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:00.455715 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 2533
2026-08-09T04:50:00Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8518 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:50:00 dist-test-slave-4tdj krb5kdc[3710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:50:00 dist-test-slave-4tdj krb5kdc[3710](info): set up 2 sockets
Aug 09 04:50:00 dist-test-slave-4tdj krb5kdc[3710](info): commencing operation
krb5kdc: starting...
W20260809 04:50:01.650352 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:50:01 dist-test-slave-4tdj krb5kdc[3710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:50:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:50:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:50:01.832964 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:45869
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:44527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:45869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:02.145496 3738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:02.145773 3738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:02.145906 3738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:02.152889 3738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:02.152992 3738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:02.153066 3738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:02.153136 3738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:02.153192 3738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:50:02.162695 3738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:44527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:45869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:45869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:02.163926 3738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:02.165206 3738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:02.176433 3744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:02.176393 3749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:02.176463 3746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:02.177188 3738 server_base.cc:1034] running on GCE node
I20260809 04:50:02.177946 3738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:02.179302 3738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:02.180505 3738 hybrid_clock.cc:648] HybridClock initialized: now 1786251002180456 us; error 38 us; skew 500 ppm
I20260809 04:50:02.180832 3738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:02 dist-test-slave-4tdj krb5kdc[3710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251002, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:02.183816 3738 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:50:02.185117 3738 webserver.cc:460] Webserver started at http://127.31.93.62:38697/ using document root <none> and password file <none>
I20260809 04:50:02.185751 3738 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:02.185849 3738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:02.186098 3738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:02.188935 3738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "5d781c738c2746fda9066f3ccd5179d3"
format_stamp: "Formatted at 2026-08-09 04:50:02 on dist-test-slave-4tdj"
server_key: "d74bc4772a2bece2668cee68936d0237"
server_key_iv: "a7175dad3830c15c1fdaadceb1302dfc"
server_key_version: "encryptionkey@0"
I20260809 04:50:02.189639 3738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "5d781c738c2746fda9066f3ccd5179d3"
format_stamp: "Formatted at 2026-08-09 04:50:02 on dist-test-slave-4tdj"
server_key: "d74bc4772a2bece2668cee68936d0237"
server_key_iv: "a7175dad3830c15c1fdaadceb1302dfc"
server_key_version: "encryptionkey@0"
I20260809 04:50:02.194495 3738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260809 04:50:02.197633 3759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:02.198917 3738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:50:02.199064 3738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "5d781c738c2746fda9066f3ccd5179d3"
format_stamp: "Formatted at 2026-08-09 04:50:02 on dist-test-slave-4tdj"
server_key: "d74bc4772a2bece2668cee68936d0237"
server_key_iv: "a7175dad3830c15c1fdaadceb1302dfc"
server_key_version: "encryptionkey@0"
I20260809 04:50:02.199237 3738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:02.211647 3738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:02.215629 3738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:02.215895 3738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:02.232582 3738 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:45869
I20260809 04:50:02.232604 3838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:45869 every 8 connection(s)
I20260809 04:50:02.233901 3738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:50:02.237279 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 3738
I20260809 04:50:02.237530 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:50:02.237568 3839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:02.238404 32116 external_mini_cluster.cc:1428] Setting key fd61ee5d0001c6c84ca6c442b947281d
I20260809 04:50:02.251207 3839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3: Bootstrap starting.
Aug 09 04:50:02 dist-test-slave-4tdj krb5kdc[3710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:02.255026 3839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:02.256191 3839 log.cc:824] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:02.259451 3839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3: No bootstrap required, opened a new log
I20260809 04:50:02.261986 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:02.240419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.31.93.62:45869)
0809 04:50:02.241182 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:02.241193 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:02.241233 (+ 40us) server_negotiation.cc:407] Connection header received
0809 04:50:02.243593 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:02.243616 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:02.243998 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:02.244315 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:02.245968 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:02.247293 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:02.249035 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:02.249701 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:02.249844 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:02.254525 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:02.254533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:02.254547 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:02.254588 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:02.257907 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:02.258745 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:02.258751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:02.258757 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:02.258921 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:02.259316 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:02.259319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:02.259321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:02.259884 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:02.260108 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:02.260580 (+ 472us) server_negotiation.cc:299] Negotiation successful
0809 04:50:02.260926 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":348,"threads_started":1}
I20260809 04:50:02.267566 3839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d781c738c2746fda9066f3ccd5179d3" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45869 } }
I20260809 04:50:02.267923 3839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:02.268016 3839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d781c738c2746fda9066f3ccd5179d3, State: Initialized, Role: FOLLOWER
I20260809 04:50:02.268592 3839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d781c738c2746fda9066f3ccd5179d3" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45869 } }
I20260809 04:50:02.268857 3839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:50:02.268982 3839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:50:02.269135 3839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:02.271287 3839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d781c738c2746fda9066f3ccd5179d3" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45869 } }
I20260809 04:50:02.271759 3839 leader_election.cc:302] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d781c738c2746fda9066f3ccd5179d3; no voters:
I20260809 04:50:02.272333 3839 leader_election.cc:288] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:50:02.272449 3849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:02.273066 3849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [term 1 LEADER]: Becoming Leader. State: Replica: 5d781c738c2746fda9066f3ccd5179d3, State: Running, Role: LEADER
I20260809 04:50:02.273567 3849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d781c738c2746fda9066f3ccd5179d3" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45869 } }
I20260809 04:50:02.274264 3839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:02.276033 3850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d781c738c2746fda9066f3ccd5179d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d781c738c2746fda9066f3ccd5179d3" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45869 } } }
I20260809 04:50:02.276451 3850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:02.277806 3851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d781c738c2746fda9066f3ccd5179d3. Latest consensus state: current_term: 1 leader_uuid: "5d781c738c2746fda9066f3ccd5179d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d781c738c2746fda9066f3ccd5179d3" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45869 } } }
I20260809 04:50:02.278043 3851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3 [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:02.278441 3860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:50:02.282917 3860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:50:02.291903 3860 catalog_manager.cc:1347] Generated new cluster ID: 06feab2d26d94127b25621f98ec59865
I20260809 04:50:02.292026 3860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:50:02.301069 3860 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:50:02.302479 3860 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:50:02.307976 3860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d781c738c2746fda9066f3ccd5179d3: Generated new TSK 0
I20260809 04:50:02.308706 3860 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:50:02.363065 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:45869
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:44527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:02.676383 3885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:02.676651 3885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:02.676782 3885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:02.684024 3885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:02.684144 3885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:02.684315 3885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:50:02.693969 3885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:44527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:45869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:02.695428 3885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:02.696767 3885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:02.709319 3892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:02.709339 3893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:02.709460 3896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:02.711501 3885 server_base.cc:1034] running on GCE node
I20260809 04:50:02.712112 3885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:02.713019 3885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:02.714219 3885 hybrid_clock.cc:648] HybridClock initialized: now 1786251002714168 us; error 44 us; skew 500 ppm
I20260809 04:50:02.714541 3885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:02 dist-test-slave-4tdj krb5kdc[3710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251002, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:02.717555 3885 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:50:02.719249 3885 webserver.cc:460] Webserver started at http://127.31.93.1:34203/ using document root <none> and password file <none>
I20260809 04:50:02.719949 3885 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:02.720065 3885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:02.720376 3885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:02.723286 3885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "20f462c6150f4d0f93fe25c9b957ebdd"
format_stamp: "Formatted at 2026-08-09 04:50:02 on dist-test-slave-4tdj"
server_key: "332b6f150f0b2e30cd4a4284b0a71ebc"
server_key_iv: "67ae1fdbc429d68bed9836e9e12ffd45"
server_key_version: "encryptionkey@0"
I20260809 04:50:02.724025 3885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "20f462c6150f4d0f93fe25c9b957ebdd"
format_stamp: "Formatted at 2026-08-09 04:50:02 on dist-test-slave-4tdj"
server_key: "332b6f150f0b2e30cd4a4284b0a71ebc"
server_key_iv: "67ae1fdbc429d68bed9836e9e12ffd45"
server_key_version: "encryptionkey@0"
I20260809 04:50:02.729121 3885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 04:50:02.732405 3906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:02.733628 3885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 04:50:02.733770 3885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "20f462c6150f4d0f93fe25c9b957ebdd"
format_stamp: "Formatted at 2026-08-09 04:50:02 on dist-test-slave-4tdj"
server_key: "332b6f150f0b2e30cd4a4284b0a71ebc"
server_key_iv: "67ae1fdbc429d68bed9836e9e12ffd45"
server_key_version: "encryptionkey@0"
I20260809 04:50:02.733949 3885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:02.745936 3885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:02.749504 3885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:02.749765 3885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:02.750748 3885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:02.752012 3885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:02.752089 3885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:02.752213 3885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:02.752272 3885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:02.782048 3885 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:33257
I20260809 04:50:02.782090 4090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:33257 every 8 connection(s)
I20260809 04:50:02.783326 3885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:50:02.790167 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 3885
I20260809 04:50:02.790429 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:50:02.791126 32116 external_mini_cluster.cc:1428] Setting key 1901453f2521041ae76068ae9a8d3496
Aug 09 04:50:02 dist-test-slave-4tdj krb5kdc[3710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251002, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:02.803931 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:02.785675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:34005 (local address 127.31.93.62:45869)
0809 04:50:02.786109 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:02.786113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:02.786969 (+ 856us) server_negotiation.cc:407] Connection header received
0809 04:50:02.788747 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:02.788753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:02.788833 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:02.788927 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:02.790709 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:02.791448 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:02.793254 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:02.793667 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:02.793759 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:02.797613 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:02.797618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:02.797622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:02.797661 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:02.800682 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:02.801527 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:02.801531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:02.801533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:02.801667 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:02.802143 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:02.802147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:02.802150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:02.802515 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:02.802631 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:02.803469 (+ 838us) server_negotiation.cc:299] Negotiation successful
0809 04:50:02.803662 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":218,"threads_started":1}
I20260809 04:50:02.805323 4097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:02.786098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45869 (local address 127.31.93.1:34005)
0809 04:50:02.786819 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:02.786863 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:02.788480 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:02.789168 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:02.789177 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:02.789678 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:02.790508 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:02.790536 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:02.791608 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:02.791614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:02.793117 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:02.793133 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:02.793877 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:02.793884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:02.794147 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:02.794848 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:02.794878 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:02.797407 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:02.800817 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:02.800826 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:02.800844 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:02.801409 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:02.801773 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:02.801778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:02.801782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:02.802009 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:02.802638 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:02.802647 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:02.803090 (+ 443us) client_negotiation.cc:769] Sending connection context
0809 04:50:02.803374 (+ 284us) client_negotiation.cc:240] Negotiation successful
0809 04:50:02.803906 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":286,"threads_started":1}
I20260809 04:50:02.806852 4091 heartbeater.cc:347] Connected to a master server at 127.31.93.62:45869
I20260809 04:50:02.807178 4091 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:02.808106 4091 heartbeater.cc:511] Master 127.31.93.62:45869 requested a full tablet report, sending...
I20260809 04:50:02.810388 3784 ts_manager.cc:194] Registered new tserver with Master: 20f462c6150f4d0f93fe25c9b957ebdd (127.31.93.1:33257)
I20260809 04:50:02.811831 3784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:34005
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:50:02.836815 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:45869
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:44527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:03.153040 4101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:03.153304 4101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:03.153439 4101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:03.160609 4101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:03.160732 4101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:03.160887 4101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:50:03.170368 4101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:44527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:45869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:03.171662 4101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:03.172976 4101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:03.185727 4109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:03.185695 4111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:03.186455 4113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:03.186842 4101 server_base.cc:1034] running on GCE node
I20260809 04:50:03.187386 4101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:03.188241 4101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:03.189421 4101 hybrid_clock.cc:648] HybridClock initialized: now 1786251003189372 us; error 49 us; skew 500 ppm
I20260809 04:50:03.189687 4101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:03 dist-test-slave-4tdj krb5kdc[3710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251003, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:03.192662 4101 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:50:03.194064 4101 webserver.cc:460] Webserver started at http://127.31.93.2:35833/ using document root <none> and password file <none>
I20260809 04:50:03.194738 4101 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:03.194852 4101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:03.195147 4101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:03.197911 4101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "4327c101ff0e41fc8d886de2b8822816"
format_stamp: "Formatted at 2026-08-09 04:50:03 on dist-test-slave-4tdj"
server_key: "f19fa42e0616964ebb5a9c6398d3fe86"
server_key_iv: "2beaff162ddd8f74539bb9fd2444becf"
server_key_version: "encryptionkey@0"
I20260809 04:50:03.198681 4101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "4327c101ff0e41fc8d886de2b8822816"
format_stamp: "Formatted at 2026-08-09 04:50:03 on dist-test-slave-4tdj"
server_key: "f19fa42e0616964ebb5a9c6398d3fe86"
server_key_iv: "2beaff162ddd8f74539bb9fd2444becf"
server_key_version: "encryptionkey@0"
I20260809 04:50:03.203732 4101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 04:50:03.207132 4125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:03.208467 4101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:50:03.208623 4101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "4327c101ff0e41fc8d886de2b8822816"
format_stamp: "Formatted at 2026-08-09 04:50:03 on dist-test-slave-4tdj"
server_key: "f19fa42e0616964ebb5a9c6398d3fe86"
server_key_iv: "2beaff162ddd8f74539bb9fd2444becf"
server_key_version: "encryptionkey@0"
I20260809 04:50:03.208801 4101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:03.223380 4101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:03.227316 4101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:03.227582 4101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:03.228482 4101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:03.229735 4101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:03.229812 4101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:03.229931 4101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:03.229990 4101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:03.263978 4101 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:43731
I20260809 04:50:03.264019 4309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:43731 every 8 connection(s)
I20260809 04:50:03.265246 4101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:50:03.274144 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 4101
I20260809 04:50:03.274361 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:50:03.274824 32116 external_mini_cluster.cc:1428] Setting key dbb58e042c3cbc649170b649b2f9d4ac
Aug 09 04:50:03 dist-test-slave-4tdj krb5kdc[3710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251003, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:03.284052 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:03.267581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:48489 (local address 127.31.93.62:45869)
0809 04:50:03.267800 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:03.267806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:03.268918 (+ 1112us) server_negotiation.cc:407] Connection header received
0809 04:50:03.270709 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:03.270713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:03.270797 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:03.270879 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:03.272224 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:03.272938 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:03.274480 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:03.274939 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:03.275066 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:03.278634 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:03.278639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:03.278642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:03.278673 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:03.281288 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:03.281938 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:03.281943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:03.281947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:03.282080 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:03.282534 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:03.282540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:03.282543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:03.282788 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:03.282914 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:03.283643 (+ 729us) server_negotiation.cc:299] Negotiation successful
0809 04:50:03.283817 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 04:50:03.285301 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:03.268055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45869 (local address 127.31.93.2:48489)
0809 04:50:03.268769 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:03.268807 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:03.270444 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:03.271057 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:03.271064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:03.271562 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:03.272097 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:03.272114 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:03.273047 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:03.273051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:03.274342 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:03.274356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:03.275149 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:03.275158 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:03.275423 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:03.276107 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:03.276129 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:03.278434 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:03.281415 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:03.281423 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:03.281438 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:03.281817 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:03.282181 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:03.282186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:03.282190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:03.282418 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:03.282909 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:03.282916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:03.283253 (+ 337us) client_negotiation.cc:769] Sending connection context
0809 04:50:03.283536 (+ 283us) client_negotiation.cc:240] Negotiation successful
0809 04:50:03.284033 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":271,"threads_started":1}
I20260809 04:50:03.286618 4310 heartbeater.cc:347] Connected to a master server at 127.31.93.62:45869
I20260809 04:50:03.286898 4310 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:03.287563 4310 heartbeater.cc:511] Master 127.31.93.62:45869 requested a full tablet report, sending...
I20260809 04:50:03.288913 3784 ts_manager.cc:194] Registered new tserver with Master: 4327c101ff0e41fc8d886de2b8822816 (127.31.93.2:43731)
I20260809 04:50:03.289675 3784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:48489
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:50:03.317147 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:45869
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:44527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:03.646787 4317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:03.647078 4317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:03.647202 4317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:03.654181 4317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:03.654322 4317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:03.654471 4317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:50:03.664448 4317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:44527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:45869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:03.665786 4317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:03.667130 4317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:03.679257 4325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:03.679334 4330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:03.679512 4326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:03.680331 4317 server_base.cc:1034] running on GCE node
I20260809 04:50:03.680917 4317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:03.681814 4317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:03.683008 4317 hybrid_clock.cc:648] HybridClock initialized: now 1786251003682949 us; error 48 us; skew 500 ppm
I20260809 04:50:03.683310 4317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:03 dist-test-slave-4tdj krb5kdc[3710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251003, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:03.686255 4317 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:50:03.687583 4317 webserver.cc:460] Webserver started at http://127.31.93.3:44629/ using document root <none> and password file <none>
I20260809 04:50:03.688197 4317 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:03.688295 4317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:03.688539 4317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:03.691342 4317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "07476bc8e5fd4ccc9d3b6fecfdd1b1fb"
format_stamp: "Formatted at 2026-08-09 04:50:03 on dist-test-slave-4tdj"
server_key: "207e3aaa6949642396450f720a9f3974"
server_key_iv: "1d424b88d94978dd536bf372c1630e07"
server_key_version: "encryptionkey@0"
I20260809 04:50:03.692087 4317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "07476bc8e5fd4ccc9d3b6fecfdd1b1fb"
format_stamp: "Formatted at 2026-08-09 04:50:03 on dist-test-slave-4tdj"
server_key: "207e3aaa6949642396450f720a9f3974"
server_key_iv: "1d424b88d94978dd536bf372c1630e07"
server_key_version: "encryptionkey@0"
I20260809 04:50:03.696867 4317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:50:03.700053 4340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:03.701273 4317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:50:03.701424 4317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "07476bc8e5fd4ccc9d3b6fecfdd1b1fb"
format_stamp: "Formatted at 2026-08-09 04:50:03 on dist-test-slave-4tdj"
server_key: "207e3aaa6949642396450f720a9f3974"
server_key_iv: "1d424b88d94978dd536bf372c1630e07"
server_key_version: "encryptionkey@0"
I20260809 04:50:03.701591 4317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:03.720358 4317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:03.724380 4317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:03.724656 4317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:03.725554 4317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:03.726856 4317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:03.726928 4317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:03.727039 4317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:03.727088 4317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:03.754550 4317 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:35225
I20260809 04:50:03.754593 4525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:35225 every 8 connection(s)
I20260809 04:50:03.755709 4317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:50:03.763080 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 4317
I20260809 04:50:03.763253 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:50:03.763643 32116 external_mini_cluster.cc:1428] Setting key 0a54108043634e09bc6f255820b5135e
Aug 09 04:50:03 dist-test-slave-4tdj krb5kdc[3710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251003, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:03.774505 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:03.758035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:49423 (local address 127.31.93.62:45869)
0809 04:50:03.758240 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:03.758246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:03.759407 (+ 1161us) server_negotiation.cc:407] Connection header received
0809 04:50:03.761131 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:03.761135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:03.761203 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:03.761288 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:03.762841 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:03.763612 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:03.765658 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:03.766321 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:03.766438 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:03.769593 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:03.769598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:03.769601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:03.769633 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:03.772014 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:03.772641 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:03.772645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:03.772647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:03.772753 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:03.773065 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:03.773068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:03.773070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:03.773314 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:03.773417 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:03.774016 (+ 599us) server_negotiation.cc:299] Negotiation successful
0809 04:50:03.774225 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 04:50:03.775393 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:03.758561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45869 (local address 127.31.93.3:49423)
0809 04:50:03.759248 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:03.759286 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:03.760872 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:03.761475 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:03.761485 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:03.762096 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:03.762703 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:03.762720 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:03.763770 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:03.763776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:03.765523 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:03.765537 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:03.766540 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:03.766548 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:03.766811 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:03.767370 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:03.767394 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:03.769402 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:03.772143 (+ 2741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:03.772156 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:03.772167 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:03.772550 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:03.772823 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:03.772826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:03.772828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:03.772990 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:03.773399 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:03.773404 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:03.773673 (+ 269us) client_negotiation.cc:769] Sending connection context
0809 04:50:03.773911 (+ 238us) client_negotiation.cc:240] Negotiation successful
0809 04:50:03.774331 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":253,"threads_started":1}
I20260809 04:50:03.776625 4526 heartbeater.cc:347] Connected to a master server at 127.31.93.62:45869
I20260809 04:50:03.776883 4526 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:03.777642 4526 heartbeater.cc:511] Master 127.31.93.62:45869 requested a full tablet report, sending...
I20260809 04:50:03.779152 3784 ts_manager.cc:194] Registered new tserver with Master: 07476bc8e5fd4ccc9d3b6fecfdd1b1fb (127.31.93.3:35225)
I20260809 04:50:03.779964 3784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:49423
I20260809 04:50:03.790599 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:50:03.814985 4091 heartbeater.cc:503] Master 127.31.93.62:45869 was elected leader, sending a full tablet report...
W20260809 04:50:03.823491 4094 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:03.821210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.31.93.62:45869)
0809 04:50:03.821398 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:03.821405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:03.821818 (+ 413us) server_negotiation.cc:407] Connection header received
0809 04:50:03.821964 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:03.821968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:03.822030 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:03.822105 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:03.823232 (+ 1127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57378: BlockingRecv error: recv got EOF from 127.0.0.1:57378 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 04:50:03.826426 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 3885
W20260809 04:50:03.828876 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 3885: Not authorized: Client connection negotiation failed: client connection to 127.31.93.1:33257: server requires authentication, but client does not have Kerberos credentials available
I20260809 04:50:03.829052 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 3885
I20260809 04:50:03.849787 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 4101
W20260809 04:50:03.852339 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 4101: Not authorized: Client connection negotiation failed: client connection to 127.31.93.2:43731: server requires authentication, but client does not have Kerberos credentials available
I20260809 04:50:03.852509 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 4101
I20260809 04:50:03.876139 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 4317
W20260809 04:50:03.879025 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 4317: Not authorized: Client connection negotiation failed: client connection to 127.31.93.3:35225: server requires authentication, but client does not have Kerberos credentials available
I20260809 04:50:03.879148 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 4317
I20260809 04:50:03.916062 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 3738
I20260809 04:50:03.917320 3830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:04.016901 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 3738
2026-08-09T04:50:04Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3558 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:50:04 dist-test-slave-4tdj krb5kdc[4554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:50:04 dist-test-slave-4tdj krb5kdc[4554](info): set up 2 sockets
Aug 09 04:50:04 dist-test-slave-4tdj krb5kdc[4554](info): commencing operation
krb5kdc: starting...
W20260809 04:50:05.230073 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260809 04:50:05.464728 32116 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.234s user 0.003s sys 0.003s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260809 04:50:05.748289 32116 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.283s user 0.000s sys 0.006s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260809 04:50:05.964850 32116 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.216s user 0.000s sys 0.006s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/krb5kdc/test-user.keytab.
W20260809 04:50:06.478242 32116 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.513s user 0.000s sys 0.007s
Aug 09 04:50:06 dist-test-slave-4tdj krb5kdc[4554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:50:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:50:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:50:06.561347 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:34015
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:34015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:06.873867 4578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:06.874150 4578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:06.874281 4578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:06.881215 4578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:06.881323 4578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:06.881402 4578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:06.881481 4578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:06.881554 4578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:50:06.891090 4578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:34015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:34015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:06.892352 4578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:06.893635 4578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:06.904664 4587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:06.904743 4588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:06.904778 4590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:06.905834 4578 server_base.cc:1034] running on GCE node
I20260809 04:50:06.906664 4578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:06.907980 4578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:06.909191 4578 hybrid_clock.cc:648] HybridClock initialized: now 1786251006909148 us; error 34 us; skew 500 ppm
I20260809 04:50:06.909529 4578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:06 dist-test-slave-4tdj krb5kdc[4554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251006, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:06.916684 4578 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:50:06.918076 4578 webserver.cc:460] Webserver started at http://127.31.93.62:36935/ using document root <none> and password file <none>
I20260809 04:50:06.918767 4578 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:06.918890 4578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:06.919191 4578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:06.922075 4578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "51cef61fca16414ab297bac7370eb43d"
format_stamp: "Formatted at 2026-08-09 04:50:06 on dist-test-slave-4tdj"
server_key: "ac6a33b1115fc6e261aa816f7aa1b213"
server_key_iv: "c9e964c3f0b06d0e29a4d8d02f9a314c"
server_key_version: "encryptionkey@0"
I20260809 04:50:06.922861 4578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "51cef61fca16414ab297bac7370eb43d"
format_stamp: "Formatted at 2026-08-09 04:50:06 on dist-test-slave-4tdj"
server_key: "ac6a33b1115fc6e261aa816f7aa1b213"
server_key_iv: "c9e964c3f0b06d0e29a4d8d02f9a314c"
server_key_version: "encryptionkey@0"
I20260809 04:50:06.927742 4578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 04:50:06.930950 4599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:06.932211 4578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:50:06.932370 4578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "51cef61fca16414ab297bac7370eb43d"
format_stamp: "Formatted at 2026-08-09 04:50:06 on dist-test-slave-4tdj"
server_key: "ac6a33b1115fc6e261aa816f7aa1b213"
server_key_iv: "c9e964c3f0b06d0e29a4d8d02f9a314c"
server_key_version: "encryptionkey@0"
I20260809 04:50:06.932542 4578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:06.957441 4578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:06.960989 4578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:06.961264 4578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:06.977913 4578 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:34015
I20260809 04:50:06.977949 4672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:34015 every 8 connection(s)
I20260809 04:50:06.979226 4578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:50:06.982823 4674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:06.986111 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 4578
I20260809 04:50:06.986373 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:50:06.987193 32116 external_mini_cluster.cc:1428] Setting key 8640199b3b75ecc84b80ab45508b9839
I20260809 04:50:06.995339 4674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d: Bootstrap starting.
I20260809 04:50:06.998958 4674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:07.000216 4674 log.cc:824] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d: Log is configured to *not* fsync() on all Append() calls
Aug 09 04:50:06 dist-test-slave-4tdj krb5kdc[4554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:07.002995 4674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d: No bootstrap required, opened a new log
I20260809 04:50:07.009212 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:06.989039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.31.93.62:34015)
0809 04:50:06.989778 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:06.989789 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:06.989830 (+ 41us) server_negotiation.cc:407] Connection header received
0809 04:50:06.991462 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:06.991487 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:06.991872 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:06.992203 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:06.994005 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:06.995305 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:06.997041 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:06.997678 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:06.997811 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:07.002378 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:07.002387 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:07.002400 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:07.002434 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:07.005330 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:07.005953 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:07.005959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:07.005964 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:07.006142 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:07.006555 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:07.006560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:07.006562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:07.007146 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:07.007452 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:07.007885 (+ 433us) server_negotiation.cc:299] Negotiation successful
0809 04:50:07.008270 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":352,"threads_started":1}
I20260809 04:50:07.010886 4674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cef61fca16414ab297bac7370eb43d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34015 } }
I20260809 04:50:07.011225 4674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:07.011328 4674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51cef61fca16414ab297bac7370eb43d, State: Initialized, Role: FOLLOWER
I20260809 04:50:07.011925 4674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cef61fca16414ab297bac7370eb43d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34015 } }
I20260809 04:50:07.012156 4674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:50:07.012308 4674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:50:07.012478 4674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:07.012779 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:50:07.014555 4674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cef61fca16414ab297bac7370eb43d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34015 } }
I20260809 04:50:07.014989 4674 leader_election.cc:302] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51cef61fca16414ab297bac7370eb43d; no voters:
I20260809 04:50:07.015542 4674 leader_election.cc:288] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:50:07.015630 4680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:07.016669 4680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [term 1 LEADER]: Becoming Leader. State: Replica: 51cef61fca16414ab297bac7370eb43d, State: Running, Role: LEADER
I20260809 04:50:07.017237 4680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cef61fca16414ab297bac7370eb43d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34015 } }
I20260809 04:50:07.017679 4674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:07.019971 4686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51cef61fca16414ab297bac7370eb43d. Latest consensus state: current_term: 1 leader_uuid: "51cef61fca16414ab297bac7370eb43d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cef61fca16414ab297bac7370eb43d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34015 } } }
I20260809 04:50:07.020300 4686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:07.025765 4684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51cef61fca16414ab297bac7370eb43d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cef61fca16414ab297bac7370eb43d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34015 } } }
I20260809 04:50:07.026085 4684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:07.029141 4692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:50:07.034545 4692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:50:07.060554 4692 catalog_manager.cc:1347] Generated new cluster ID: b6b784ec909c4045b578c5faab145d09
I20260809 04:50:07.060683 4692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:50:07.077768 4692 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:50:07.079886 4692 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:50:07.083364 4728 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 04:50:07.093636 4692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51cef61fca16414ab297bac7370eb43d: Generated new TSK 0
I20260809 04:50:07.094671 4692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:50:07.095006 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:07.085670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39836 (local address 127.31.93.62:34015)
0809 04:50:07.085830 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:07.085836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:07.086412 (+ 576us) server_negotiation.cc:407] Connection header received
0809 04:50:07.086593 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:07.086598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:07.086700 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:07.086812 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:07.087954 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:07.089106 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:07.090945 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:07.091588 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:07.091719 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:07.092936 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:07.092943 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:07.092948 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:07.094012 (+ 1064us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:07.094092 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:07.094119 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:07.094225 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:07.094461 (+ 236us) server_negotiation.cc:299] Negotiation successful
0809 04:50:07.094566 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 04:50:07.105182 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 4578
I20260809 04:50:07.106438 4666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:07.202962 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 4578
2026-08-09T04:50:07Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (3182 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:50:07 dist-test-slave-4tdj krb5kdc[4735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:50:07 dist-test-slave-4tdj krb5kdc[4735](info): set up 2 sockets
Aug 09 04:50:07 dist-test-slave-4tdj krb5kdc[4735](info): commencing operation
krb5kdc: starting...
W20260809 04:50:08.388288 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:50:08 dist-test-slave-4tdj krb5kdc[4735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:50:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:50:08.571020 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:39723
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:39723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:08.888295 4761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:08.888568 4761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:08.888664 4761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:08.895622 4761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:08.895814 4761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:08.895908 4761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:08.896050 4761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:08.896121 4761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:50:08.905742 4761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:39723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:39723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:08.907063 4761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:08.908337 4761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:08.919582 4771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:08.919492 4772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:08.919464 4777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:08.920401 4761 server_base.cc:1034] running on GCE node
I20260809 04:50:08.921083 4761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:08.922356 4761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:08.923583 4761 hybrid_clock.cc:648] HybridClock initialized: now 1786251008923531 us; error 35 us; skew 500 ppm
I20260809 04:50:08.923904 4761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:08 dist-test-slave-4tdj krb5kdc[4735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251008, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:08.927038 4761 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:50:08.928438 4761 webserver.cc:460] Webserver started at http://127.31.93.62:43601/ using document root <none> and password file <none>
I20260809 04:50:08.929104 4761 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:08.929237 4761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:08.929534 4761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:08.932363 4761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "2b0b5626c8e342dba11e45cdbe578576"
format_stamp: "Formatted at 2026-08-09 04:50:08 on dist-test-slave-4tdj"
server_key: "d1b96fc36947b7d3ad7a9724bcd61844"
server_key_iv: "326e3561518bc7041054d1d3f3c387f4"
server_key_version: "encryptionkey@0"
I20260809 04:50:08.933113 4761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "2b0b5626c8e342dba11e45cdbe578576"
format_stamp: "Formatted at 2026-08-09 04:50:08 on dist-test-slave-4tdj"
server_key: "d1b96fc36947b7d3ad7a9724bcd61844"
server_key_iv: "326e3561518bc7041054d1d3f3c387f4"
server_key_version: "encryptionkey@0"
I20260809 04:50:08.937988 4761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 04:50:08.941253 4785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:08.942523 4761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:50:08.942677 4761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "2b0b5626c8e342dba11e45cdbe578576"
format_stamp: "Formatted at 2026-08-09 04:50:08 on dist-test-slave-4tdj"
server_key: "d1b96fc36947b7d3ad7a9724bcd61844"
server_key_iv: "326e3561518bc7041054d1d3f3c387f4"
server_key_version: "encryptionkey@0"
I20260809 04:50:08.942842 4761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:08.960075 4761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:08.963745 4761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:08.963994 4761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:08.980964 4761 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:39723
I20260809 04:50:08.980988 4857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:39723 every 8 connection(s)
I20260809 04:50:08.982266 4761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:50:08.985849 4858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:08.988703 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 4761
I20260809 04:50:08.988929 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:50:08.989758 32116 external_mini_cluster.cc:1428] Setting key fb9345e9436d9df98750bd0e96fc326e
I20260809 04:50:08.997958 4858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576: Bootstrap starting.
I20260809 04:50:09.001575 4858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:09.002859 4858 log.cc:824] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576: Log is configured to *not* fsync() on all Append() calls
Aug 09 04:50:08 dist-test-slave-4tdj krb5kdc[4735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:09.005887 4858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576: No bootstrap required, opened a new log
I20260809 04:50:09.011670 4862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:08.991785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48160 (local address 127.31.93.62:39723)
0809 04:50:08.992510 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:08.992524 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:08.992564 (+ 40us) server_negotiation.cc:407] Connection header received
0809 04:50:08.994145 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:08.994167 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:08.994595 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:08.994914 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:08.996513 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:08.997768 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:08.999453 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:09.000123 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:09.000247 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:09.004944 (+ 4697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:09.004951 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:09.004964 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:09.005006 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:09.007878 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:09.008617 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:09.008623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:09.008628 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:09.008784 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:09.009148 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:09.009152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:09.009154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:09.009652 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:09.009868 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:09.010341 (+ 473us) server_negotiation.cc:299] Negotiation successful
0809 04:50:09.010711 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":339,"threads_started":1}
I20260809 04:50:09.014487 4858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0b5626c8e342dba11e45cdbe578576" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 39723 } }
I20260809 04:50:09.014817 4858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:09.014899 4858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0b5626c8e342dba11e45cdbe578576, State: Initialized, Role: FOLLOWER
I20260809 04:50:09.015414 4858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0b5626c8e342dba11e45cdbe578576" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 39723 } }
I20260809 04:50:09.015630 4858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:50:09.015753 4858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:50:09.015903 4858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:09.018045 4858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0b5626c8e342dba11e45cdbe578576" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 39723 } }
I20260809 04:50:09.018488 4858 leader_election.cc:302] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b0b5626c8e342dba11e45cdbe578576; no voters:
I20260809 04:50:09.018978 4858 leader_election.cc:288] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:50:09.019093 4867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:09.020202 4867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [term 1 LEADER]: Becoming Leader. State: Replica: 2b0b5626c8e342dba11e45cdbe578576, State: Running, Role: LEADER
I20260809 04:50:09.020947 4858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:09.020843 4867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0b5626c8e342dba11e45cdbe578576" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 39723 } }
I20260809 04:50:09.025417 4868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b0b5626c8e342dba11e45cdbe578576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0b5626c8e342dba11e45cdbe578576" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 39723 } } }
I20260809 04:50:09.025812 4868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:09.026788 4870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b0b5626c8e342dba11e45cdbe578576. Latest consensus state: current_term: 1 leader_uuid: "2b0b5626c8e342dba11e45cdbe578576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0b5626c8e342dba11e45cdbe578576" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 39723 } } }
I20260809 04:50:09.027046 4875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:50:09.027123 4870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576 [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:09.033000 4875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:50:09.041155 4875 catalog_manager.cc:1347] Generated new cluster ID: 3978f87f574f4de6b07f795475a1cde0
I20260809 04:50:09.041258 4875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:50:09.079912 4875 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:50:09.081314 4875 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:50:09.097239 4875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b0b5626c8e342dba11e45cdbe578576: Generated new TSK 0
I20260809 04:50:09.098039 4875 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:50:09.151073 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:39723
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:09.463670 4897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:09.463919 4897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:09.464017 4897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:09.471055 4897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:09.471159 4897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:09.471273 4897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:50:09.480528 4897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:39723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:09.481686 4897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:09.482942 4897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:09.495276 4902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:09.495222 4908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:09.495189 4904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:09.495977 4897 server_base.cc:1034] running on GCE node
I20260809 04:50:09.496515 4897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:09.497350 4897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:09.498544 4897 hybrid_clock.cc:648] HybridClock initialized: now 1786251009498486 us; error 48 us; skew 500 ppm
I20260809 04:50:09.498828 4897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:09 dist-test-slave-4tdj krb5kdc[4735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251009, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:09.501705 4897 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:50:09.503041 4897 webserver.cc:460] Webserver started at http://127.31.93.1:39275/ using document root <none> and password file <none>
I20260809 04:50:09.503672 4897 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:09.503786 4897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:09.504056 4897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:09.506819 4897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "1bfbb21310df44b9acc6950f1544bf98"
format_stamp: "Formatted at 2026-08-09 04:50:09 on dist-test-slave-4tdj"
server_key: "55da4b31ce3fa099ef8c565fec345c0d"
server_key_iv: "f2689b078d4df89e9f46358025e773ae"
server_key_version: "encryptionkey@0"
I20260809 04:50:09.507553 4897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "1bfbb21310df44b9acc6950f1544bf98"
format_stamp: "Formatted at 2026-08-09 04:50:09 on dist-test-slave-4tdj"
server_key: "55da4b31ce3fa099ef8c565fec345c0d"
server_key_iv: "f2689b078d4df89e9f46358025e773ae"
server_key_version: "encryptionkey@0"
I20260809 04:50:09.512468 4897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 04:50:09.515673 4922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:09.516861 4897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:50:09.517012 4897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "1bfbb21310df44b9acc6950f1544bf98"
format_stamp: "Formatted at 2026-08-09 04:50:09 on dist-test-slave-4tdj"
server_key: "55da4b31ce3fa099ef8c565fec345c0d"
server_key_iv: "f2689b078d4df89e9f46358025e773ae"
server_key_version: "encryptionkey@0"
I20260809 04:50:09.517184 4897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:09.530179 4897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:09.533568 4897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:09.533818 4897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:09.534679 4897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:09.535864 4897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:09.535945 4897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:09.536057 4897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:09.536123 4897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:09.564812 4897 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:46791
I20260809 04:50:09.564857 5134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:46791 every 8 connection(s)
I20260809 04:50:09.565987 4897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:50:09.569094 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 4897
I20260809 04:50:09.569314 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:50:09.570109 32116 external_mini_cluster.cc:1428] Setting key 7ff0611be4158ab3c5a67c75c61e7627
Aug 09 04:50:09 dist-test-slave-4tdj krb5kdc[4735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251009, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:09.584821 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:09.568298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:32855 (local address 127.31.93.62:39723)
0809 04:50:09.568728 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:09.568733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:09.569755 (+ 1022us) server_negotiation.cc:407] Connection header received
0809 04:50:09.571671 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:09.571677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:09.571759 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:09.571851 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:09.573273 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:09.574004 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:09.575540 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:09.575945 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:09.576057 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:09.579347 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:09.579353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:09.579356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:09.579387 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:09.581982 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:09.582791 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:09.582795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:09.582798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:09.582912 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:09.583331 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:09.583335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:09.583337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:09.583587 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:09.583698 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:09.584410 (+ 712us) server_negotiation.cc:299] Negotiation successful
0809 04:50:09.584564 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":235,"threads_started":1}
I20260809 04:50:09.586174 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:09.568771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:39723 (local address 127.31.93.1:32855)
0809 04:50:09.569603 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:09.569649 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:09.571403 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:09.572084 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:09.572092 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:09.572577 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:09.573112 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:09.573129 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:09.574148 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:09.574152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:09.575402 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:09.575414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:09.576174 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:09.576181 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:09.576391 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:09.576985 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:09.577007 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:09.579138 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:09.582125 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:09.582133 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:09.582147 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:09.582686 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:09.583010 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:09.583014 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:09.583017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:09.583244 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:09.583716 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:09.583723 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:09.584088 (+ 365us) client_negotiation.cc:769] Sending connection context
0809 04:50:09.584349 (+ 261us) client_negotiation.cc:240] Negotiation successful
0809 04:50:09.584862 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":343,"threads_started":1}
I20260809 04:50:09.587620 5135 heartbeater.cc:347] Connected to a master server at 127.31.93.62:39723
I20260809 04:50:09.587949 5135 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:09.588851 5135 heartbeater.cc:511] Master 127.31.93.62:39723 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
I20260809 04:50:09.591082 4807 ts_manager.cc:194] Registered new tserver with Master: 1bfbb21310df44b9acc6950f1544bf98 (127.31.93.1:46791)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
I20260809 04:50:09.592603 4807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:32855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:50:09.613260 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:39723
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:09.923970 5147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:09.924233 5147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:09.924371 5147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:09.931401 5147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:09.931516 5147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:09.931663 5147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:50:09.941180 5147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:39723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:09.942387 5147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:09.943670 5147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:09.955966 5156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:09.956017 5159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:09.956130 5153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:09.956907 5147 server_base.cc:1034] running on GCE node
I20260809 04:50:09.957455 5147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:09.958375 5147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:09.959592 5147 hybrid_clock.cc:648] HybridClock initialized: now 1786251009959514 us; error 64 us; skew 500 ppm
I20260809 04:50:09.959877 5147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:09 dist-test-slave-4tdj krb5kdc[4735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251009, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:09.962788 5147 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:50:09.964130 5147 webserver.cc:460] Webserver started at http://127.31.93.2:38725/ using document root <none> and password file <none>
I20260809 04:50:09.964730 5147 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:09.964828 5147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:09.965086 5147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:09.967867 5147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "4239ccdb73a64525976069a121d2796d"
format_stamp: "Formatted at 2026-08-09 04:50:09 on dist-test-slave-4tdj"
server_key: "98039364a94c27e6bb56bfe165f1527a"
server_key_iv: "9b90b53a757a1d53122f594ead36adc3"
server_key_version: "encryptionkey@0"
I20260809 04:50:09.968636 5147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "4239ccdb73a64525976069a121d2796d"
format_stamp: "Formatted at 2026-08-09 04:50:09 on dist-test-slave-4tdj"
server_key: "98039364a94c27e6bb56bfe165f1527a"
server_key_iv: "9b90b53a757a1d53122f594ead36adc3"
server_key_version: "encryptionkey@0"
I20260809 04:50:09.973466 5147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 04:50:09.976682 5170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:09.977891 5147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 04:50:09.978035 5147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "4239ccdb73a64525976069a121d2796d"
format_stamp: "Formatted at 2026-08-09 04:50:09 on dist-test-slave-4tdj"
server_key: "98039364a94c27e6bb56bfe165f1527a"
server_key_iv: "9b90b53a757a1d53122f594ead36adc3"
server_key_version: "encryptionkey@0"
I20260809 04:50:09.978211 5147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:10.000581 5147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:10.004528 5147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:10.004797 5147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:10.005756 5147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:10.007090 5147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:10.007174 5147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:10.007290 5147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:10.007340 5147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:10.041302 5147 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:37339
I20260809 04:50:10.041348 5361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:37339 every 8 connection(s)
I20260809 04:50:10.042603 5147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:50:10.049539 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 5147
I20260809 04:50:10.049700 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:50:10.050246 32116 external_mini_cluster.cc:1428] Setting key b229b94e83660dcc917c95cb4fdb7850
Aug 09 04:50:10 dist-test-slave-4tdj krb5kdc[4735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251009, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:10.062552 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:10.044948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:54531 (local address 127.31.93.62:39723)
0809 04:50:10.045170 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:10.045176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:10.046430 (+ 1254us) server_negotiation.cc:407] Connection header received
0809 04:50:10.048208 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:10.048213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:10.048297 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:10.048391 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:10.049899 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:10.050922 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:10.052914 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:10.053320 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:10.053414 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:10.057183 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:10.057189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:10.057192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:10.057224 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:10.059666 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:10.060339 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:10.060344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:10.060346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:10.060480 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:10.060892 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:10.060895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:10.060898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:10.061145 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:10.061293 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:10.061972 (+ 679us) server_negotiation.cc:299] Negotiation successful
0809 04:50:10.062207 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 04:50:10.063514 5368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:10.045473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:39723 (local address 127.31.93.2:54531)
0809 04:50:10.046264 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:10.046335 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:10.047987 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:10.048569 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:10.048577 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:10.049097 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:10.049678 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:10.049700 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:10.051081 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:10.051087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:10.052775 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:10.052790 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:10.053541 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:10.053548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:10.053815 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:10.054541 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:10.054569 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:10.056971 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:10.059795 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:10.059804 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:10.059821 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:10.060226 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:10.060584 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:10.060590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:10.060594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:10.060786 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:10.061277 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:10.061285 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:10.061597 (+ 312us) client_negotiation.cc:769] Sending connection context
0809 04:50:10.061875 (+ 278us) client_negotiation.cc:240] Negotiation successful
0809 04:50:10.062404 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":314,"threads_started":1}
I20260809 04:50:10.064934 5366 heartbeater.cc:347] Connected to a master server at 127.31.93.62:39723
I20260809 04:50:10.065335 5366 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:10.066149 5366 heartbeater.cc:511] Master 127.31.93.62:39723 requested a full tablet report, sending...
I20260809 04:50:10.067763 4807 ts_manager.cc:194] Registered new tserver with Master: 4239ccdb73a64525976069a121d2796d (127.31.93.2:37339)
I20260809 04:50:10.068885 4807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:54531
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:50:10.095203 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:39723
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:10.404471 5371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:10.404731 5371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:10.404831 5371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:10.411744 5371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:10.411837 5371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:10.411954 5371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:50:10.421244 5371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:39723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:10.422516 5371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:10.423785 5371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:10.435973 5377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:10.436033 5378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:10.435999 5381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:10.436710 5371 server_base.cc:1034] running on GCE node
I20260809 04:50:10.437234 5371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:10.438052 5371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:10.439241 5371 hybrid_clock.cc:648] HybridClock initialized: now 1786251010439191 us; error 45 us; skew 500 ppm
I20260809 04:50:10.439540 5371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:10 dist-test-slave-4tdj krb5kdc[4735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251010, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:10.442447 5371 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:50:10.443766 5371 webserver.cc:460] Webserver started at http://127.31.93.3:43069/ using document root <none> and password file <none>
I20260809 04:50:10.444379 5371 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:10.444489 5371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:10.444783 5371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:10.447727 5371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "5a0f2fd3bc114a32a0237ca5fc9ad30d"
format_stamp: "Formatted at 2026-08-09 04:50:10 on dist-test-slave-4tdj"
server_key: "4e428ddba201403a38b10d990d070899"
server_key_iv: "85c1d526c678613c3afefab82c3423ad"
server_key_version: "encryptionkey@0"
I20260809 04:50:10.448459 5371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "5a0f2fd3bc114a32a0237ca5fc9ad30d"
format_stamp: "Formatted at 2026-08-09 04:50:10 on dist-test-slave-4tdj"
server_key: "4e428ddba201403a38b10d990d070899"
server_key_iv: "85c1d526c678613c3afefab82c3423ad"
server_key_version: "encryptionkey@0"
I20260809 04:50:10.453212 5371 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260809 04:50:10.456466 5391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:10.457684 5371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 04:50:10.457823 5371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "5a0f2fd3bc114a32a0237ca5fc9ad30d"
format_stamp: "Formatted at 2026-08-09 04:50:10 on dist-test-slave-4tdj"
server_key: "4e428ddba201403a38b10d990d070899"
server_key_iv: "85c1d526c678613c3afefab82c3423ad"
server_key_version: "encryptionkey@0"
I20260809 04:50:10.457983 5371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:10.474527 5371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:10.478022 5371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:10.478274 5371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:10.479177 5371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:10.480425 5371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:10.480496 5371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:10.480619 5371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:10.480671 5371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:10.509753 5371 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:33643
I20260809 04:50:10.509764 5563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:33643 every 8 connection(s)
I20260809 04:50:10.510979 5371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:50:10.520048 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 5371
I20260809 04:50:10.520233 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:50:10.520718 32116 external_mini_cluster.cc:1428] Setting key 6468a7f1882b6a10129b27b3272d22b3
Aug 09 04:50:10 dist-test-slave-4tdj krb5kdc[4735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251010, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:10.529553 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:10.513312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:47399 (local address 127.31.93.62:39723)
0809 04:50:10.513477 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:10.513481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:10.514604 (+ 1123us) server_negotiation.cc:407] Connection header received
0809 04:50:10.516321 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:10.516327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:10.516405 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:10.516491 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:10.517926 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:10.518701 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:10.520241 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:10.520719 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:10.520832 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:10.524436 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:10.524442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:10.524445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:10.524476 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:10.526730 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:10.527381 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:10.527387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:10.527390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:10.527507 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:10.527923 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:10.527928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:10.527932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:10.528194 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:10.528333 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:10.528976 (+ 643us) server_negotiation.cc:299] Negotiation successful
0809 04:50:10.529202 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:50:10.530452 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:10.513729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:39723 (local address 127.31.93.3:47399)
0809 04:50:10.514456 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:10.514501 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:10.516051 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:10.516732 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:10.516740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:10.517234 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:10.517762 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:10.517779 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:10.518845 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:10.518849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:10.520108 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:10.520120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:10.520947 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:10.520955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:10.521219 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:10.521910 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:10.521948 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:10.524236 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:10.526853 (+ 2617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:10.526862 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:10.526877 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:10.527269 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:10.527611 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:10.527616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:10.527619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:10.527798 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:10.528317 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:10.528324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:10.528647 (+ 323us) client_negotiation.cc:769] Sending connection context
0809 04:50:10.528878 (+ 231us) client_negotiation.cc:240] Negotiation successful
0809 04:50:10.529358 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":272,"threads_started":1}
I20260809 04:50:10.531580 5564 heartbeater.cc:347] Connected to a master server at 127.31.93.62:39723
I20260809 04:50:10.531814 5564 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:10.532467 5564 heartbeater.cc:511] Master 127.31.93.62:39723 requested a full tablet report, sending...
I20260809 04:50:10.533691 4807 ts_manager.cc:194] Registered new tserver with Master: 5a0f2fd3bc114a32a0237ca5fc9ad30d (127.31.93.3:33643)
I20260809 04:50:10.534679 4807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:47399
I20260809 04:50:10.535769 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 04:50:10 dist-test-slave-4tdj krb5kdc[4735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251010, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 09 04:50:10 dist-test-slave-4tdj krb5kdc[4735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251010, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:10.585343 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:10.572423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48170 (local address 127.31.93.62:39723)
0809 04:50:10.572580 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:10.572585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:10.573063 (+ 478us) server_negotiation.cc:407] Connection header received
0809 04:50:10.573207 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:10.573211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:10.573271 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:10.573355 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:10.574658 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:10.575350 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:10.576486 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:10.576860 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:10.576951 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:10.580436 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:10.580441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:10.580444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:10.580477 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:10.582913 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:10.583498 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:10.583504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:10.583507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:10.583622 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:10.583993 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:10.583999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:10.584002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:10.584304 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:10.584541 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:10.584835 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 04:50:10.585039 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260809 04:50:10.585831 4807 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48170
I20260809 04:50:10.587503 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 4897
I20260809 04:50:10.595458 5135 heartbeater.cc:503] Master 127.31.93.62:39723 was elected leader, sending a full tablet report...
Aug 09 04:50:10 dist-test-slave-4tdj krb5kdc[4735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251010, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:50:10.601390 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:10.588420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35788 (local address 127.31.93.1:46791)
0809 04:50:10.588989 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:10.588996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:10.589019 (+ 23us) server_negotiation.cc:407] Connection header received
0809 04:50:10.589069 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:10.589075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:10.589266 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:10.589380 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:10.590437 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:10.591400 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:10.592465 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:10.593003 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:10.593092 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:10.596710 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:10.596724 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:10.596740 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:10.596779 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:10.599312 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:10.599874 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:10.599880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:10.599882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:10.599990 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:10.600264 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:10.600267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:10.600269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:10.600566 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:10.600786 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:10.600945 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 04:50:10.601124 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":295,"threads_started":1}
W20260809 04:50:10.602567 5127 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35788
W20260809 04:50:10.603564 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 4897: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:50:10.603760 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 4897
I20260809 04:50:10.624900 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 5147
Aug 09 04:50:10 dist-test-slave-4tdj krb5kdc[4735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251010, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:50:10.639636 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:10.625817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.31.93.2:37339)
0809 04:50:10.626370 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:10.626377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:10.626403 (+ 26us) server_negotiation.cc:407] Connection header received
0809 04:50:10.626463 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:10.626470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:10.626697 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:10.626819 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:10.628053 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:10.629091 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:10.630314 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:10.630899 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:10.630996 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:10.634835 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:10.634847 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:10.634861 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:10.634903 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:10.637341 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:10.637893 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:10.637900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:10.637902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:10.638022 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:10.638339 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:10.638343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:10.638346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:10.638684 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:10.638907 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:10.639141 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 04:50:10.639338 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":260,"threads_started":1}
W20260809 04:50:10.640849 5354 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36788
W20260809 04:50:10.641886 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 5147: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:50:10.642023 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 5147
I20260809 04:50:10.664458 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 5371
Aug 09 04:50:10 dist-test-slave-4tdj krb5kdc[4735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251010, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:50:10.679260 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:10.665377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49794 (local address 127.31.93.3:33643)
0809 04:50:10.666022 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:10.666029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:10.666058 (+ 29us) server_negotiation.cc:407] Connection header received
0809 04:50:10.666131 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:10.666137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:10.666375 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:10.666508 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:10.667688 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:10.668705 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:10.669812 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:10.670458 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:10.670551 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:10.674551 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:10.674563 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:10.674577 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:10.674611 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:10.677007 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:10.677584 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:10.677590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:10.677592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:10.677709 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:10.677969 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:10.677972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:10.677974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:10.678356 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:10.678580 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:10.678789 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 04:50:10.678975 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":332,"threads_started":1}
W20260809 04:50:10.680444 5557 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49794
W20260809 04:50:10.681442 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 5371: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:50:10.681593 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 5371
I20260809 04:50:10.710194 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 4761
I20260809 04:50:10.711359 4852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:10.811646 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 4761
2026-08-09T04:50:10Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3613 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:50:10 dist-test-slave-4tdj krb5kdc[5589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:50:10 dist-test-slave-4tdj krb5kdc[5589](info): set up 2 sockets
Aug 09 04:50:10 dist-test-slave-4tdj krb5kdc[5589](info): commencing operation
krb5kdc: starting...
W20260809 04:50:12.009503 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:50:12 dist-test-slave-4tdj krb5kdc[5589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:50:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:50:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:50:12.192853 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:46745
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:46745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:12.501871 5604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:12.502135 5604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:12.502223 5604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:12.509151 5604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:12.509248 5604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:12.509298 5604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:12.509353 5604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:12.509397 5604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:50:12.518993 5604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:46745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:46745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:12.520113 5604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:12.521346 5604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:12.532491 5612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:12.532506 5613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:12.532588 5615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:12.533402 5604 server_base.cc:1034] running on GCE node
I20260809 04:50:12.534150 5604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:12.535503 5604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:12.536705 5604 hybrid_clock.cc:648] HybridClock initialized: now 1786251012536657 us; error 41 us; skew 500 ppm
I20260809 04:50:12.537022 5604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:12 dist-test-slave-4tdj krb5kdc[5589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251012, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:12.544096 5604 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:50:12.545495 5604 webserver.cc:460] Webserver started at http://127.31.93.62:44141/ using document root <none> and password file <none>
I20260809 04:50:12.546089 5604 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:12.546219 5604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:12.546537 5604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:12.549261 5604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "7d8818bb6518436689612bd6503e764f"
format_stamp: "Formatted at 2026-08-09 04:50:12 on dist-test-slave-4tdj"
server_key: "6e94092f3ba55916e0302e88b815b5cd"
server_key_iv: "49a022a1727d9157c1687ae20a3ea25e"
server_key_version: "encryptionkey@0"
I20260809 04:50:12.549963 5604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "7d8818bb6518436689612bd6503e764f"
format_stamp: "Formatted at 2026-08-09 04:50:12 on dist-test-slave-4tdj"
server_key: "6e94092f3ba55916e0302e88b815b5cd"
server_key_iv: "49a022a1727d9157c1687ae20a3ea25e"
server_key_version: "encryptionkey@0"
I20260809 04:50:12.554672 5604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260809 04:50:12.557870 5625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:12.559116 5604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:50:12.559255 5604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "7d8818bb6518436689612bd6503e764f"
format_stamp: "Formatted at 2026-08-09 04:50:12 on dist-test-slave-4tdj"
server_key: "6e94092f3ba55916e0302e88b815b5cd"
server_key_iv: "49a022a1727d9157c1687ae20a3ea25e"
server_key_version: "encryptionkey@0"
I20260809 04:50:12.559420 5604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:12.594934 5604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:12.598584 5604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:12.598840 5604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:12.615509 5604 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:46745
I20260809 04:50:12.615536 5717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:46745 every 8 connection(s)
I20260809 04:50:12.616778 5604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:50:12.617369 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 5604
I20260809 04:50:12.617576 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:50:12.618376 32116 external_mini_cluster.cc:1428] Setting key 44be2305118f733cca1a04a2923f9fe7
I20260809 04:50:12.620992 5718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 04:50:12 dist-test-slave-4tdj krb5kdc[5589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:12.633903 5718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f: Bootstrap starting.
I20260809 04:50:12.637845 5718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:12.639026 5718 log.cc:824] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:12.641950 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:12.620265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44664 (local address 127.31.93.62:46745)
0809 04:50:12.621074 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:12.621088 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:12.621137 (+ 49us) server_negotiation.cc:407] Connection header received
0809 04:50:12.623376 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:12.623403 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:12.623882 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:12.624301 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:12.625571 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:12.627609 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:12.628749 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:12.629719 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:12.629910 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:12.634086 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:12.634096 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:12.634113 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:12.634154 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:12.637739 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:12.638387 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:12.638394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:12.638401 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:12.638609 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:12.638927 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:12.638931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:12.638934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:12.639603 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:12.639852 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:12.640300 (+ 448us) server_negotiation.cc:299] Negotiation successful
0809 04:50:12.640715 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":314,"threads_started":1}
I20260809 04:50:12.642199 5718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f: No bootstrap required, opened a new log
I20260809 04:50:12.649866 5718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8818bb6518436689612bd6503e764f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 46745 } }
I20260809 04:50:12.650187 5718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:12.650257 5718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d8818bb6518436689612bd6503e764f, State: Initialized, Role: FOLLOWER
I20260809 04:50:12.650799 5718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8818bb6518436689612bd6503e764f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 46745 } }
I20260809 04:50:12.651005 5718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:50:12.651104 5718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:50:12.651218 5718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:12.653106 5718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8818bb6518436689612bd6503e764f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 46745 } }
I20260809 04:50:12.653523 5718 leader_election.cc:302] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d8818bb6518436689612bd6503e764f; no voters:
I20260809 04:50:12.654004 5718 leader_election.cc:288] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:50:12.654129 5731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:12.654745 5731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [term 1 LEADER]: Becoming Leader. State: Replica: 7d8818bb6518436689612bd6503e764f, State: Running, Role: LEADER
I20260809 04:50:12.655301 5731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8818bb6518436689612bd6503e764f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 46745 } }
I20260809 04:50:12.655830 5718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:12.657869 5732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d8818bb6518436689612bd6503e764f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8818bb6518436689612bd6503e764f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 46745 } } }
I20260809 04:50:12.657902 5736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d8818bb6518436689612bd6503e764f. Latest consensus state: current_term: 1 leader_uuid: "7d8818bb6518436689612bd6503e764f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8818bb6518436689612bd6503e764f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 46745 } } }
I20260809 04:50:12.658550 5732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:12.658602 5736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:12.659523 5741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:50:12.664733 5741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:50:12.673768 5741 catalog_manager.cc:1347] Generated new cluster ID: 349efb191d3c445c9ac2a0082ecf7fed
I20260809 04:50:12.673874 5741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:50:12.686113 5741 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:50:12.687397 5741 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:50:12.693950 5741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d8818bb6518436689612bd6503e764f: Generated new TSK 0
I20260809 04:50:12.694640 5741 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:50:12.745998 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:46745
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:13.066711 5761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:13.066963 5761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:13.067063 5761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:13.073998 5761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:13.074092 5761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:13.074208 5761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:50:13.083351 5761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:46745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:13.084421 5761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:13.085656 5761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:13.097463 5768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:13.097573 5769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:13.097499 5771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:13.098623 5761 server_base.cc:1034] running on GCE node
I20260809 04:50:13.099138 5761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:13.099928 5761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:13.101123 5761 hybrid_clock.cc:648] HybridClock initialized: now 1786251013101083 us; error 27 us; skew 500 ppm
I20260809 04:50:13.101384 5761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:13 dist-test-slave-4tdj krb5kdc[5589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251013, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:13.104233 5761 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:50:13.105507 5761 webserver.cc:460] Webserver started at http://127.31.93.1:45833/ using document root <none> and password file <none>
I20260809 04:50:13.106117 5761 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:13.106235 5761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:13.106566 5761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:13.109263 5761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "a7018ac93d384f04a3eb8fff199944e8"
format_stamp: "Formatted at 2026-08-09 04:50:13 on dist-test-slave-4tdj"
server_key: "b32be24f7d62bb2100ed8c8bffcb9f4c"
server_key_iv: "267eaa60be09b959522daca7aafd59a2"
server_key_version: "encryptionkey@0"
I20260809 04:50:13.110002 5761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "a7018ac93d384f04a3eb8fff199944e8"
format_stamp: "Formatted at 2026-08-09 04:50:13 on dist-test-slave-4tdj"
server_key: "b32be24f7d62bb2100ed8c8bffcb9f4c"
server_key_iv: "267eaa60be09b959522daca7aafd59a2"
server_key_version: "encryptionkey@0"
I20260809 04:50:13.114786 5761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 04:50:13.117967 5779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:13.119159 5761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:50:13.119313 5761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "a7018ac93d384f04a3eb8fff199944e8"
format_stamp: "Formatted at 2026-08-09 04:50:13 on dist-test-slave-4tdj"
server_key: "b32be24f7d62bb2100ed8c8bffcb9f4c"
server_key_iv: "267eaa60be09b959522daca7aafd59a2"
server_key_version: "encryptionkey@0"
I20260809 04:50:13.119485 5761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:13.142495 5761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:13.146087 5761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:13.146373 5761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:13.147261 5761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:13.148484 5761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:13.148561 5761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:13.148698 5761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:13.148752 5761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:13.178020 5761 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:41365
I20260809 04:50:13.178083 5966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:41365 every 8 connection(s)
I20260809 04:50:13.179214 5761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:50:13.184207 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 5761
I20260809 04:50:13.184433 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:50:13.185267 32116 external_mini_cluster.cc:1428] Setting key 9901c8655748910b2ac7a6a1d5e1b566
Aug 09 04:50:13 dist-test-slave-4tdj krb5kdc[5589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251013, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:13.200179 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:13.181456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:51507 (local address 127.31.93.62:46745)
0809 04:50:13.181863 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:13.181867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:13.182693 (+ 826us) server_negotiation.cc:407] Connection header received
0809 04:50:13.184781 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:13.184786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:13.184856 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:13.184951 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:13.186709 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:13.187621 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:13.189778 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:13.190313 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:13.190420 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:13.194252 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:13.194258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:13.194263 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:13.194320 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:13.197294 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:13.197940 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:13.197946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:13.197949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:13.198072 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:13.198517 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:13.198522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:13.198526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:13.198801 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:13.198933 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:13.199576 (+ 643us) server_negotiation.cc:299] Negotiation successful
0809 04:50:13.199818 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":223,"threads_started":1}
I20260809 04:50:13.201009 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:13.181846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:46745 (local address 127.31.93.1:51507)
0809 04:50:13.182544 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:13.182589 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:13.184473 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:13.185233 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:13.185245 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:13.185933 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:13.186537 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:13.186556 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:13.187779 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:13.187785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:13.189638 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:13.189654 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:13.190522 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:13.190528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:13.190790 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:13.191462 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:13.191488 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:13.194015 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:13.197424 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:13.197431 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:13.197442 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:13.197829 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:13.198183 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:13.198186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:13.198189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:13.198411 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:13.198933 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:13.198941 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:13.199240 (+ 299us) client_negotiation.cc:769] Sending connection context
0809 04:50:13.199458 (+ 218us) client_negotiation.cc:240] Negotiation successful
0809 04:50:13.199912 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":258,"threads_started":1}
I20260809 04:50:13.202471 5967 heartbeater.cc:347] Connected to a master server at 127.31.93.62:46745
I20260809 04:50:13.202745 5967 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:13.203374 5967 heartbeater.cc:511] Master 127.31.93.62:46745 requested a full tablet report, sending...
I20260809 04:50:13.205299 5652 ts_manager.cc:194] Registered new tserver with Master: a7018ac93d384f04a3eb8fff199944e8 (127.31.93.1:41365)
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
I20260809 04:50:13.206740 5652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:51507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:50:13.230083 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:46745
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:13.551579 5973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:13.551828 5973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:13.551959 5973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:13.558880 5973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:13.558977 5973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:13.559135 5973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:50:13.568257 5973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:46745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:13.569402 5973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:13.570701 5973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:13.582901 5982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:13.582864 5986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:13.582937 5984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:13.583871 5973 server_base.cc:1034] running on GCE node
I20260809 04:50:13.584488 5973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:13.585354 5973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:13.586544 5973 hybrid_clock.cc:648] HybridClock initialized: now 1786251013586499 us; error 32 us; skew 500 ppm
I20260809 04:50:13.586827 5973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:13 dist-test-slave-4tdj krb5kdc[5589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251013, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:13.589655 5973 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:50:13.591037 5973 webserver.cc:460] Webserver started at http://127.31.93.2:46329/ using document root <none> and password file <none>
I20260809 04:50:13.591660 5973 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:13.591776 5973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:13.592058 5973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:13.594844 5973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "58f355dc75784ba58fe1dc9aff417721"
format_stamp: "Formatted at 2026-08-09 04:50:13 on dist-test-slave-4tdj"
server_key: "a6781a8471b667e5aafc98ef73fdb230"
server_key_iv: "2890b9b11fc22c47ef15edbb4c06c164"
server_key_version: "encryptionkey@0"
I20260809 04:50:13.595582 5973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "58f355dc75784ba58fe1dc9aff417721"
format_stamp: "Formatted at 2026-08-09 04:50:13 on dist-test-slave-4tdj"
server_key: "a6781a8471b667e5aafc98ef73fdb230"
server_key_iv: "2890b9b11fc22c47ef15edbb4c06c164"
server_key_version: "encryptionkey@0"
I20260809 04:50:13.600324 5973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:50:13.603551 5993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:13.604745 5973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:50:13.604882 5973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "58f355dc75784ba58fe1dc9aff417721"
format_stamp: "Formatted at 2026-08-09 04:50:13 on dist-test-slave-4tdj"
server_key: "a6781a8471b667e5aafc98ef73fdb230"
server_key_iv: "2890b9b11fc22c47ef15edbb4c06c164"
server_key_version: "encryptionkey@0"
I20260809 04:50:13.605050 5973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:13.627359 5973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:13.631999 5973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:13.632332 5973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:13.633330 5973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:13.634857 5973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:13.634949 5973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:13.635074 5973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:13.635138 5973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:13.671490 5973 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:33549
I20260809 04:50:13.671535 6169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:33549 every 8 connection(s)
I20260809 04:50:13.672669 5973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:50:13.675554 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 5973
I20260809 04:50:13.675724 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:50:13.676132 32116 external_mini_cluster.cc:1428] Setting key 8c5230ae5b9c4dcf80d6b2c559d7981a
Aug 09 04:50:13 dist-test-slave-4tdj krb5kdc[5589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251013, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:13.691661 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:13.675081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:38143 (local address 127.31.93.62:46745)
0809 04:50:13.675250 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:13.675256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:13.676453 (+ 1197us) server_negotiation.cc:407] Connection header received
0809 04:50:13.678327 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:13.678333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:13.678416 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:13.678521 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:13.679898 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:13.680841 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:13.682367 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:13.682925 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:13.683046 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:13.686325 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:13.686332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:13.686338 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:13.686372 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:13.688758 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:13.689436 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:13.689441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:13.689444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:13.689573 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:13.689981 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:13.689985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:13.689988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:13.690252 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:13.690419 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:13.691083 (+ 664us) server_negotiation.cc:299] Negotiation successful
0809 04:50:13.691313 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 04:50:13.692627 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:13.675500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:46745 (local address 127.31.93.2:38143)
0809 04:50:13.676287 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:13.676340 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:13.678055 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:13.678709 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:13.678717 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:13.679196 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:13.679731 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:13.679749 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:13.680998 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:13.681002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:13.682208 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:13.682220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:13.683124 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:13.683129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:13.683323 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:13.683916 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:13.683946 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:13.686084 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:13.688888 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:13.688897 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:13.688913 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:13.689321 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:13.689686 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:13.689692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:13.689696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:13.689874 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:13.690401 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:13.690409 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:13.690725 (+ 316us) client_negotiation.cc:769] Sending connection context
0809 04:50:13.690964 (+ 239us) client_negotiation.cc:240] Negotiation successful
0809 04:50:13.691475 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":315,"threads_started":1}
I20260809 04:50:13.694144 6170 heartbeater.cc:347] Connected to a master server at 127.31.93.62:46745
I20260809 04:50:13.694489 6170 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:13.695183 6170 heartbeater.cc:511] Master 127.31.93.62:46745 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
I20260809 04:50:13.696619 5652 ts_manager.cc:194] Registered new tserver with Master: 58f355dc75784ba58fe1dc9aff417721 (127.31.93.2:33549)
I20260809 04:50:13.697502 5652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:38143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:50:13.720679 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:46745
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:14.045713 6176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:14.046185 6176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:14.046434 6176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:14.057731 6176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:14.057866 6176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:14.058065 6176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:50:14.073504 6176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:46745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:14.075502 6176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:14.077389 6176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:14.090166 6186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:14.090816 6183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:14.090934 6189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:14.091791 6176 server_base.cc:1034] running on GCE node
I20260809 04:50:14.092514 6176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:14.093642 6176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:14.094921 6176 hybrid_clock.cc:648] HybridClock initialized: now 1786251014094817 us; error 81 us; skew 500 ppm
I20260809 04:50:14.095301 6176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:14 dist-test-slave-4tdj krb5kdc[5589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251014, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:14.099280 6176 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:50:14.101006 6176 webserver.cc:460] Webserver started at http://127.31.93.3:33403/ using document root <none> and password file <none>
I20260809 04:50:14.101861 6176 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:14.102006 6176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:14.102402 6176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:14.106618 6176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "363c9e8dd2d540dbb310bc42255811a5"
format_stamp: "Formatted at 2026-08-09 04:50:14 on dist-test-slave-4tdj"
server_key: "5b14443812f90187ca322bbb47735b6a"
server_key_iv: "5215980847534f3084f984ff38480047"
server_key_version: "encryptionkey@0"
I20260809 04:50:14.107679 6176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "363c9e8dd2d540dbb310bc42255811a5"
format_stamp: "Formatted at 2026-08-09 04:50:14 on dist-test-slave-4tdj"
server_key: "5b14443812f90187ca322bbb47735b6a"
server_key_iv: "5215980847534f3084f984ff38480047"
server_key_version: "encryptionkey@0"
I20260809 04:50:14.114426 6176 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260809 04:50:14.118798 6198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:14.120376 6176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 04:50:14.120545 6176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "363c9e8dd2d540dbb310bc42255811a5"
format_stamp: "Formatted at 2026-08-09 04:50:14 on dist-test-slave-4tdj"
server_key: "5b14443812f90187ca322bbb47735b6a"
server_key_iv: "5215980847534f3084f984ff38480047"
server_key_version: "encryptionkey@0"
I20260809 04:50:14.120784 6176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:14.148916 6176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:14.153311 6176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:14.153604 6176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:14.154556 6176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:14.155858 6176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:14.155973 6176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:14.156087 6176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:14.156153 6176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:14.193174 6176 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:36717
I20260809 04:50:14.193253 6380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:36717 every 8 connection(s)
I20260809 04:50:14.194489 6176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:50:14.197621 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 6176
I20260809 04:50:14.197806 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:50:14.198392 32116 external_mini_cluster.cc:1428] Setting key 713e6e1238d32bade01801916d597140
Aug 09 04:50:14 dist-test-slave-4tdj krb5kdc[5589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251014, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:14.209462 5967 heartbeater.cc:503] Master 127.31.93.62:46745 was elected leader, sending a full tablet report...
I20260809 04:50:14.216387 6384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:14.197163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:33583 (local address 127.31.93.62:46745)
0809 04:50:14.197673 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:14.197680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:14.201395 (+ 3715us) server_negotiation.cc:407] Connection header received
0809 04:50:14.201464 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:14.201469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:14.201556 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:14.201683 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:14.203116 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:14.204345 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:14.205843 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:14.206510 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:14.206644 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:14.210145 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:14.210152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:14.210156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:14.210199 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:14.213528 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:14.214171 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:14.214177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:14.214181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:14.214346 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:14.214762 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:14.214767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:14.214771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:14.215030 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:14.215176 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:14.215815 (+ 639us) server_negotiation.cc:299] Negotiation successful
0809 04:50:14.216020 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":533,"threads_started":1}
I20260809 04:50:14.217298 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:14.197688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:46745 (local address 127.31.93.3:33583)
0809 04:50:14.198616 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:14.198677 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:14.201060 (+ 2383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:14.201870 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:14.201877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:14.202412 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:14.202953 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:14.202970 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:14.204471 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:14.204475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:14.205701 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:14.205712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:14.206698 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:14.206703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:14.206897 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:14.207543 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:14.207563 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:14.209935 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:14.213641 (+ 3706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:14.213650 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:14.213666 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:14.214059 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:14.214459 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:14.214464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:14.214468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:14.214656 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:14.215160 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:14.215167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:14.215471 (+ 304us) client_negotiation.cc:769] Sending connection context
0809 04:50:14.215710 (+ 239us) client_negotiation.cc:240] Negotiation successful
0809 04:50:14.216186 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":337,"threads_started":1}
I20260809 04:50:14.218535 6381 heartbeater.cc:347] Connected to a master server at 127.31.93.62:46745
I20260809 04:50:14.218873 6381 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:14.219761 6381 heartbeater.cc:511] Master 127.31.93.62:46745 requested a full tablet report, sending...
I20260809 04:50:14.221277 5652 ts_manager.cc:194] Registered new tserver with Master: 363c9e8dd2d540dbb310bc42255811a5 (127.31.93.3:36717)
I20260809 04:50:14.222035 5652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:33583
I20260809 04:50:14.225360 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 04:50:14 dist-test-slave-4tdj krb5kdc[5589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 04:50:14 dist-test-slave-4tdj krb5kdc[5589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:50:14.274722 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:14.260856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.31.93.1:41365)
0809 04:50:14.261503 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:14.261510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:14.261553 (+ 43us) server_negotiation.cc:407] Connection header received
0809 04:50:14.261631 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:14.261637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:14.261837 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:14.261975 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:14.263371 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:14.264444 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:14.265601 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:14.266143 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:14.266236 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:14.269768 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:14.269778 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:14.269792 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:14.269828 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:14.272469 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:14.272980 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:14.272986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:14.272988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:14.273105 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:14.273369 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:14.273372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:14.273374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:14.273694 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:14.273906 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:14.274102 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 04:50:14.274366 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":334,"threads_started":1}
Aug 09 04:50:14 dist-test-slave-4tdj krb5kdc[5589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:14.291369 6384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:14.278690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.31.93.62:46745)
0809 04:50:14.278890 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:14.278895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:14.279060 (+ 165us) server_negotiation.cc:407] Connection header received
0809 04:50:14.279217 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:14.279222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:14.279291 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:14.279371 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:14.280554 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:14.281270 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:14.282538 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:14.282940 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:14.283051 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:14.286728 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:14.286733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:14.286736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:14.286768 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:14.288915 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:14.289489 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:14.289495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:14.289498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:14.289614 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:14.289964 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:14.289969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:14.289972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:14.290341 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:14.290565 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:14.290870 (+ 305us) server_negotiation.cc:299] Negotiation successful
0809 04:50:14.291084 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260809 04:50:14.291889 5652 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44670
I20260809 04:50:14.292788 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 5761
I20260809 04:50:14.303802 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:14.293669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35590 (local address 127.31.93.1:41365)
0809 04:50:14.293855 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:14.293861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:14.293967 (+ 106us) server_negotiation.cc:407] Connection header received
0809 04:50:14.294096 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:14.294100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:14.294215 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:14.294342 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:14.295377 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:14.296005 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:14.297054 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:14.297448 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:14.297535 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:14.299436 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:14.299440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:14.299443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:14.299470 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:14.301846 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:14.302383 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:14.302386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:14.302388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:14.302497 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:14.302777 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:14.302780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:14.302782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:14.302991 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:14.303151 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:14.303360 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 04:50:14.303509 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 04:50:14.304359 5961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:14.388129 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 5761
I20260809 04:50:14.405927 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 5973
Aug 09 04:50:14 dist-test-slave-4tdj krb5kdc[5589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:50:14.420714 6402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:14.406993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35424 (local address 127.31.93.2:33549)
0809 04:50:14.407530 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:14.407537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:14.407562 (+ 25us) server_negotiation.cc:407] Connection header received
0809 04:50:14.407626 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:14.407632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:14.407841 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:14.407969 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:14.409179 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:14.410178 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:14.411396 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:14.411935 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:14.412032 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:14.415930 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:14.415940 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:14.415953 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:14.415989 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:14.418540 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:14.419093 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:14.419099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:14.419102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:14.419228 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:14.419502 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:14.419505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:14.419507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:14.419811 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:14.420025 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:14.420246 (+ 221us) server_negotiation.cc:299] Negotiation successful
0809 04:50:14.420434 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":258,"threads_started":1}
I20260809 04:50:14.421907 6165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:14.501116 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 5973
I20260809 04:50:14.522217 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 6176
Aug 09 04:50:14 dist-test-slave-4tdj krb5kdc[5589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:50:14.537257 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:14.523180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.31.93.3:36717)
0809 04:50:14.523804 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:14.523811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:14.523842 (+ 31us) server_negotiation.cc:407] Connection header received
0809 04:50:14.523906 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:14.523912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:14.524122 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:14.524259 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:14.525538 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:14.526818 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:14.528038 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:14.528585 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:14.528677 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:14.532610 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:14.532621 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:14.532636 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:14.532670 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:14.535135 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:14.535691 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:14.535697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:14.535700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:14.535817 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:14.536073 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:14.536075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:14.536077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:14.536386 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:14.536620 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:14.536795 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 04:50:14.536984 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":290,"threads_started":1}
I20260809 04:50:14.538479 6370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:14.617906 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 6176
I20260809 04:50:14.638041 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 5604
I20260809 04:50:14.639201 5712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:14.740039 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 5604
2026-08-09T04:50:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3927 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:50:14 dist-test-slave-4tdj krb5kdc[6410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:50:14 dist-test-slave-4tdj krb5kdc[6410](info): set up 2 sockets
Aug 09 04:50:14 dist-test-slave-4tdj krb5kdc[6410](info): commencing operation
krb5kdc: starting...
W20260809 04:50:15.922547 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:50:16 dist-test-slave-4tdj krb5kdc[6410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:50:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:50:16.105700 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:33249
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:33249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:16.411017 6429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:16.411293 6429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:16.411428 6429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:16.418267 6429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:16.418391 6429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:16.418471 6429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:16.418550 6429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:16.418615 6429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:50:16.427994 6429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:33249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:33249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:16.429204 6429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:16.430485 6429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:16.441200 6436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:16.441149 6437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:16.441237 6440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:16.441931 6429 server_base.cc:1034] running on GCE node
I20260809 04:50:16.442663 6429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:16.443857 6429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:16.445046 6429 hybrid_clock.cc:648] HybridClock initialized: now 1786251016445007 us; error 27 us; skew 500 ppm
I20260809 04:50:16.445382 6429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:16 dist-test-slave-4tdj krb5kdc[6410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251016, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:16.448221 6429 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:50:16.449249 6429 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:16.449352 6429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:16.449656 6429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:16.452452 6429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "fc3562bda7a04743bb2d1e50c61cf786"
format_stamp: "Formatted at 2026-08-09 04:50:16 on dist-test-slave-4tdj"
server_key: "f810904e33b34da39f1af583894af409"
server_key_iv: "2aa7a0f5a11df9c3c930a0a970ad420e"
server_key_version: "encryptionkey@0"
I20260809 04:50:16.453189 6429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "fc3562bda7a04743bb2d1e50c61cf786"
format_stamp: "Formatted at 2026-08-09 04:50:16 on dist-test-slave-4tdj"
server_key: "f810904e33b34da39f1af583894af409"
server_key_iv: "2aa7a0f5a11df9c3c930a0a970ad420e"
server_key_version: "encryptionkey@0"
I20260809 04:50:16.457754 6429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 04:50:16.460948 6450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:16.462097 6429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:50:16.462257 6429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "fc3562bda7a04743bb2d1e50c61cf786"
format_stamp: "Formatted at 2026-08-09 04:50:16 on dist-test-slave-4tdj"
server_key: "f810904e33b34da39f1af583894af409"
server_key_iv: "2aa7a0f5a11df9c3c930a0a970ad420e"
server_key_version: "encryptionkey@0"
I20260809 04:50:16.462483 6429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:16.474512 6429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:16.477854 6429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:16.478080 6429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:16.494091 6429 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:33249
I20260809 04:50:16.494110 6524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:33249 every 8 connection(s)
I20260809 04:50:16.494886 6429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:50:16.498095 6525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:16.499640 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 6429
I20260809 04:50:16.499836 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:50:16.500445 32116 external_mini_cluster.cc:1428] Setting key d23aba6419996789b530dfa9a360de23
I20260809 04:50:16.510434 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786: Bootstrap starting.
I20260809 04:50:16.513760 6525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786: Neither blocks nor log segments found. Creating new log.
Aug 09 04:50:16 dist-test-slave-4tdj krb5kdc[6410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:16.514765 6525 log.cc:824] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:16.517766 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786: No bootstrap required, opened a new log
I20260809 04:50:16.523205 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:16.502197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.31.93.62:33249)
0809 04:50:16.503059 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:16.503077 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:16.503129 (+ 52us) server_negotiation.cc:407] Connection header received
0809 04:50:16.505362 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:16.505389 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:16.505859 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:16.506251 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:16.507593 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:16.509506 (+ 1913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:16.510758 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:16.511742 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:16.511940 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:16.515482 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:16.515489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:16.515502 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:16.515534 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:16.519050 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:16.519668 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:16.519676 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:16.519684 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:16.519893 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:16.520203 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:16.520208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:16.520212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:16.520915 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:16.521221 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:16.521504 (+ 283us) server_negotiation.cc:299] Negotiation successful
0809 04:50:16.521919 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":333,"threads_started":1}
I20260809 04:50:16.527176 6525 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3562bda7a04743bb2d1e50c61cf786" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33249 } }
I20260809 04:50:16.527484 6525 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:16.527554 6525 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc3562bda7a04743bb2d1e50c61cf786, State: Initialized, Role: FOLLOWER
I20260809 04:50:16.528008 6525 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3562bda7a04743bb2d1e50c61cf786" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33249 } }
I20260809 04:50:16.528203 6525 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:50:16.528298 6525 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:50:16.528405 6525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:16.530426 6525 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3562bda7a04743bb2d1e50c61cf786" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33249 } }
I20260809 04:50:16.530936 6525 leader_election.cc:302] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc3562bda7a04743bb2d1e50c61cf786; no voters:
I20260809 04:50:16.531528 6525 leader_election.cc:288] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:50:16.531601 6535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:16.532233 6535 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [term 1 LEADER]: Becoming Leader. State: Replica: fc3562bda7a04743bb2d1e50c61cf786, State: Running, Role: LEADER
I20260809 04:50:16.532693 6535 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3562bda7a04743bb2d1e50c61cf786" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33249 } }
I20260809 04:50:16.533380 6525 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:16.535353 6536 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc3562bda7a04743bb2d1e50c61cf786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3562bda7a04743bb2d1e50c61cf786" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33249 } } }
I20260809 04:50:16.535703 6536 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:16.535331 6537 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc3562bda7a04743bb2d1e50c61cf786. Latest consensus state: current_term: 1 leader_uuid: "fc3562bda7a04743bb2d1e50c61cf786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc3562bda7a04743bb2d1e50c61cf786" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33249 } } }
I20260809 04:50:16.536890 6537 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786 [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:16.537058 6542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:50:16.541735 6542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:50:16.549917 6542 catalog_manager.cc:1347] Generated new cluster ID: cd0061c62a8e4e28b55fe4be331802f2
I20260809 04:50:16.550015 6542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:50:16.561630 6542 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:50:16.562821 6542 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:50:16.570369 6542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786: Generated new TSK 0
I20260809 04:50:16.571009 6542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:50:16.619529 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33249
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:16.933600 6565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:16.933877 6565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:16.933987 6565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:16.941078 6565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:16.941170 6565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:16.941293 6565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:50:16.950439 6565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:16.951552 6565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:16.952759 6565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:16.964424 6576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:16.964504 6573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:16.964455 6574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:16.965389 6565 server_base.cc:1034] running on GCE node
I20260809 04:50:16.965946 6565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:16.966996 6565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:16.968207 6565 hybrid_clock.cc:648] HybridClock initialized: now 1786251016968154 us; error 36 us; skew 500 ppm
I20260809 04:50:16.968503 6565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:16 dist-test-slave-4tdj krb5kdc[6410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251016, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:16.971594 6565 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:50:16.972645 6565 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:16.972747 6565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:16.973043 6565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:16.976143 6565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "b265d37162724b7cb471071d71d88e36"
format_stamp: "Formatted at 2026-08-09 04:50:16 on dist-test-slave-4tdj"
server_key: "879cf58f2f94a506f081d7a20d01df58"
server_key_iv: "a92665027d368f0b640536d21ff97691"
server_key_version: "encryptionkey@0"
I20260809 04:50:16.976998 6565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "b265d37162724b7cb471071d71d88e36"
format_stamp: "Formatted at 2026-08-09 04:50:16 on dist-test-slave-4tdj"
server_key: "879cf58f2f94a506f081d7a20d01df58"
server_key_iv: "a92665027d368f0b640536d21ff97691"
server_key_version: "encryptionkey@0"
I20260809 04:50:16.981757 6565 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:50:16.985118 6588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:16.986375 6565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:50:16.986514 6565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "b265d37162724b7cb471071d71d88e36"
format_stamp: "Formatted at 2026-08-09 04:50:16 on dist-test-slave-4tdj"
server_key: "879cf58f2f94a506f081d7a20d01df58"
server_key_iv: "a92665027d368f0b640536d21ff97691"
server_key_version: "encryptionkey@0"
I20260809 04:50:16.986724 6565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:17.001729 6565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:17.005782 6565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:17.006026 6565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:17.006930 6565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:17.008150 6565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:17.008242 6565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:17.008342 6565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:17.008419 6565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:17.041787 6565 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:32909
I20260809 04:50:17.041847 6746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:32909 every 8 connection(s)
I20260809 04:50:17.042595 6565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:50:17.046476 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 6565
I20260809 04:50:17.046712 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:50:17.047478 32116 external_mini_cluster.cc:1428] Setting key adb6dfa505be8f2cdaabfd88272bf572
Aug 09 04:50:17 dist-test-slave-4tdj krb5kdc[6410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251016, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:17.061445 6755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:17.044821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:42417 (local address 127.31.93.62:33249)
0809 04:50:17.045247 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:17.045252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:17.046103 (+ 851us) server_negotiation.cc:407] Connection header received
0809 04:50:17.048428 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:17.048434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:17.048515 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:17.048640 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:17.050197 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:17.050974 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:17.052460 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:17.052841 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:17.052949 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:17.056144 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:17.056149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:17.056152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:17.056183 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:17.058611 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:17.059266 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:17.059270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:17.059273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:17.059390 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:17.059787 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:17.059793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:17.059795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:17.060061 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:17.060206 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:17.060853 (+ 647us) server_negotiation.cc:299] Negotiation successful
0809 04:50:17.061080 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":224,"threads_started":1}
I20260809 04:50:17.062248 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:17.045261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:33249 (local address 127.31.93.1:42417)
0809 04:50:17.045928 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:17.045965 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:17.048175 (+ 2210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:17.048866 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:17.048874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:17.049486 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:17.050026 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:17.050050 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:17.051116 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:17.051120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:17.052320 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:17.052333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:17.053049 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:17.053057 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:17.053277 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:17.053848 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:17.053870 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:17.055945 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:17.058738 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:17.058746 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:17.058761 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:17.059154 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:17.059493 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:17.059497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:17.059502 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:17.059681 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:17.060184 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:17.060192 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:17.060485 (+ 293us) client_negotiation.cc:769] Sending connection context
0809 04:50:17.060742 (+ 257us) client_negotiation.cc:240] Negotiation successful
0809 04:50:17.061206 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":269,"threads_started":1}
I20260809 04:50:17.063680 6751 heartbeater.cc:347] Connected to a master server at 127.31.93.62:33249
I20260809 04:50:17.063942 6751 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:17.064539 6751 heartbeater.cc:511] Master 127.31.93.62:33249 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
I20260809 04:50:17.066632 6476 ts_manager.cc:194] Registered new tserver with Master: b265d37162724b7cb471071d71d88e36 (127.31.93.1:32909)
I20260809 04:50:17.068145 6476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:42417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:50:17.089303 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33249
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:17.399452 6762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:17.399722 6762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:17.399856 6762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:17.406826 6762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:17.406919 6762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:17.407068 6762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:50:17.416177 6762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:17.417346 6762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:17.418588 6762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:17.430215 6769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:17.430660 6773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:17.430685 6768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:17.431139 6762 server_base.cc:1034] running on GCE node
I20260809 04:50:17.431685 6762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:17.432505 6762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:17.433696 6762 hybrid_clock.cc:648] HybridClock initialized: now 1786251017433645 us; error 45 us; skew 500 ppm
I20260809 04:50:17.433991 6762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:17 dist-test-slave-4tdj krb5kdc[6410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251017, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:17.436916 6762 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:50:17.437906 6762 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:17.437990 6762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:17.438244 6762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:17.440982 6762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "08381773767546648fe66e52759ecf4e"
format_stamp: "Formatted at 2026-08-09 04:50:17 on dist-test-slave-4tdj"
server_key: "3c0b13998fd064553e7437fe7d478368"
server_key_iv: "a90e06e6caf311980c4a90ff51730668"
server_key_version: "encryptionkey@0"
I20260809 04:50:17.441752 6762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "08381773767546648fe66e52759ecf4e"
format_stamp: "Formatted at 2026-08-09 04:50:17 on dist-test-slave-4tdj"
server_key: "3c0b13998fd064553e7437fe7d478368"
server_key_iv: "a90e06e6caf311980c4a90ff51730668"
server_key_version: "encryptionkey@0"
I20260809 04:50:17.446394 6762 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.007s
I20260809 04:50:17.449555 6782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:17.450773 6762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:50:17.450922 6762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "08381773767546648fe66e52759ecf4e"
format_stamp: "Formatted at 2026-08-09 04:50:17 on dist-test-slave-4tdj"
server_key: "3c0b13998fd064553e7437fe7d478368"
server_key_iv: "a90e06e6caf311980c4a90ff51730668"
server_key_version: "encryptionkey@0"
I20260809 04:50:17.451116 6762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:17.498771 6762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:17.502713 6762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:17.502952 6762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:17.503859 6762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:17.505120 6762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:17.505213 6762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:17.505328 6762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:17.505398 6762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:17.534852 6762 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:42287
I20260809 04:50:17.534937 6961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:42287 every 8 connection(s)
I20260809 04:50:17.535645 6762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:50:17.545212 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 6762
I20260809 04:50:17.545418 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:50:17.545956 32116 external_mini_cluster.cc:1428] Setting key 162139b3a5fa4e7f145e1dd4576da942
Aug 09 04:50:17 dist-test-slave-4tdj krb5kdc[6410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251017, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:17.558185 6755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:17.537975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:49811 (local address 127.31.93.62:33249)
0809 04:50:17.538178 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:17.538184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:17.539369 (+ 1185us) server_negotiation.cc:407] Connection header received
0809 04:50:17.541204 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:17.541209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:17.541281 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:17.541356 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:17.542759 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:17.543466 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:17.545033 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:17.545659 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:17.545779 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:17.552355 (+ 6576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:17.552361 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:17.552365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:17.552400 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:17.555288 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:17.556141 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:17.556145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:17.556147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:17.556275 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:17.556705 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:17.556708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:17.556710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:17.556948 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:17.557062 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:17.557790 (+ 728us) server_negotiation.cc:299] Negotiation successful
0809 04:50:17.557948 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 04:50:17.559489 6965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:17.538470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:33249 (local address 127.31.93.2:49811)
0809 04:50:17.539209 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:17.539245 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:17.540959 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:17.541534 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:17.541542 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:17.542027 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:17.542619 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:17.542637 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:17.543569 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:17.543572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:17.544895 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:17.544908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:17.545850 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:17.545855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:17.546044 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:17.549942 (+ 3898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:17.549964 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:17.552143 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:17.555410 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:17.555418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:17.555434 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:17.556026 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:17.556368 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:17.556372 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:17.556375 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:17.556599 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:17.557077 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:17.557084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:17.557444 (+ 360us) client_negotiation.cc:769] Sending connection context
0809 04:50:17.557725 (+ 281us) client_negotiation.cc:240] Negotiation successful
0809 04:50:17.558182 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":276,"threads_started":1}
I20260809 04:50:17.560956 6963 heartbeater.cc:347] Connected to a master server at 127.31.93.62:33249
I20260809 04:50:17.561234 6963 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:17.561815 6963 heartbeater.cc:511] Master 127.31.93.62:33249 requested a full tablet report, sending...
I20260809 04:50:17.563207 6476 ts_manager.cc:194] Registered new tserver with Master: 08381773767546648fe66e52759ecf4e (127.31.93.2:42287)
I20260809 04:50:17.563963 6476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:49811
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:50:17.590930 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33249
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:17.900022 6970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:17.900303 6970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:17.900449 6970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:17.907321 6970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:17.907429 6970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:17.907586 6970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:50:17.916756 6970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:17.917961 6970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:17.919220 6970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:17.930778 6978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:17.930825 6980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:17.930851 6982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:17.931735 6970 server_base.cc:1034] running on GCE node
I20260809 04:50:17.932273 6970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:17.933099 6970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:17.934322 6970 hybrid_clock.cc:648] HybridClock initialized: now 1786251017934236 us; error 56 us; skew 500 ppm
I20260809 04:50:17.934609 6970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:17 dist-test-slave-4tdj krb5kdc[6410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251017, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:17.937387 6970 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:50:17.938465 6970 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:17.938565 6970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:17.938853 6970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:17.941605 6970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "1013419418b0403b85312e4e2c664320"
format_stamp: "Formatted at 2026-08-09 04:50:17 on dist-test-slave-4tdj"
server_key: "5b5df8036ffabc6e87927bb1ae202bb9"
server_key_iv: "d6987c1f44fbe7c6b2e9670fb3154a5d"
server_key_version: "encryptionkey@0"
I20260809 04:50:17.942373 6970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "1013419418b0403b85312e4e2c664320"
format_stamp: "Formatted at 2026-08-09 04:50:17 on dist-test-slave-4tdj"
server_key: "5b5df8036ffabc6e87927bb1ae202bb9"
server_key_iv: "d6987c1f44fbe7c6b2e9670fb3154a5d"
server_key_version: "encryptionkey@0"
I20260809 04:50:17.946889 6970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 04:50:17.950004 6999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:17.951246 6970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:50:17.951396 6970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "1013419418b0403b85312e4e2c664320"
format_stamp: "Formatted at 2026-08-09 04:50:17 on dist-test-slave-4tdj"
server_key: "5b5df8036ffabc6e87927bb1ae202bb9"
server_key_iv: "d6987c1f44fbe7c6b2e9670fb3154a5d"
server_key_version: "encryptionkey@0"
I20260809 04:50:17.951584 6970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:17.969188 6970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:17.972877 6970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:17.973107 6970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:17.973981 6970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:17.975382 6970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:17.975476 6970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:17.975579 6970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:17.975646 6970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:18.005378 6970 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:41199
I20260809 04:50:18.005406 7174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:41199 every 8 connection(s)
I20260809 04:50:18.006245 6970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:50:18.015646 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 6970
I20260809 04:50:18.015837 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:50:18.016232 32116 external_mini_cluster.cc:1428] Setting key 7177d22945d09644adb8519b840a0193
Aug 09 04:50:18 dist-test-slave-4tdj krb5kdc[6410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251017, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:18.024600 6755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.008695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:47917 (local address 127.31.93.62:33249)
0809 04:50:18.008875 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:18.008880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:18.009995 (+ 1115us) server_negotiation.cc:407] Connection header received
0809 04:50:18.011810 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:18.011816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:18.011895 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:18.011981 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:18.013485 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.014218 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.015756 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.016231 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.016343 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.019843 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:18.019848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:18.019851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:18.019883 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:18.022195 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:18.022857 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:18.022861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:18.022863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:18.022983 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:18.023303 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:18.023306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:18.023308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:18.023563 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:18.023671 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:18.024220 (+ 549us) server_negotiation.cc:299] Negotiation successful
0809 04:50:18.024370 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:50:18.025534 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.009130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:33249 (local address 127.31.93.3:47917)
0809 04:50:18.009840 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:18.009877 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:18.011550 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:18.012234 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:18.012243 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:18.012760 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:18.013321 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:18.013338 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.014398 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.014403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:18.015616 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:18.015628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.016452 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.016460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:18.016673 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.017286 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:18.017317 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:18.019644 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:18.022341 (+ 2697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:18.022347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:18.022358 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:18.022772 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:18.023061 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:18.023065 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:18.023067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:18.023224 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:18.023655 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:18.023661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:18.023928 (+ 267us) client_negotiation.cc:769] Sending connection context
0809 04:50:18.024137 (+ 209us) client_negotiation.cc:240] Negotiation successful
0809 04:50:18.024485 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":289,"threads_started":1}
I20260809 04:50:18.026825 7175 heartbeater.cc:347] Connected to a master server at 127.31.93.62:33249
I20260809 04:50:18.027097 7175 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:18.027694 7175 heartbeater.cc:511] Master 127.31.93.62:33249 requested a full tablet report, sending...
I20260809 04:50:18.029057 6476 ts_manager.cc:194] Registered new tserver with Master: 1013419418b0403b85312e4e2c664320 (127.31.93.3:41199)
I20260809 04:50:18.029938 6476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:47917
I20260809 04:50:18.031280 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:50:18.048017 6755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.035639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.31.93.62:33249)
0809 04:50:18.035774 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:18.035779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:18.036268 (+ 489us) server_negotiation.cc:407] Connection header received
0809 04:50:18.036417 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:18.036422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:18.036478 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:18.036553 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:18.037913 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.038644 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.039976 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.040431 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.040531 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.042764 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:18.042770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:18.042774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:18.042811 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:18.045822 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:18.046457 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:18.046462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:18.046465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:18.046606 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:18.046900 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:18.046904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:18.046906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:18.047251 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:18.047433 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:18.047659 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 04:50:18.047803 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 04:50:18.051441 6476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:50:18.058068 6476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:50:18.071007 6751 heartbeater.cc:503] Master 127.31.93.62:33249 was elected leader, sending a full tablet report...
I20260809 04:50:18.079305 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.069308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51592 (local address 127.31.93.3:41199)
0809 04:50:18.070285 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:18.070322 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:18.070461 (+ 139us) server_negotiation.cc:407] Connection header received
0809 04:50:18.070705 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:18.070713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:18.070967 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:18.071171 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:18.072179 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.073864 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.077230 (+ 3366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.078143 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.078263 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.078523 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:18.078802 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 04:50:18.078965 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":492,"threads_started":1}
I20260809 04:50:18.079399 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.069020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:41199 (local address 127.0.0.1:51592)
0809 04:50:18.070327 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:18.070368 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:18.070513 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:18.071484 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:18.071493 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:18.071530 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:18.071946 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:18.071961 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.074002 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.074007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:18.077087 (+ 3080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:18.077106 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.078353 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.078360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:18.078571 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.078626 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 04:50:18.078757 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 04:50:18.078949 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"mutex_wait_us":5,"thread_start_us":940,"threads_started":1}
I20260809 04:50:18.081672 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.070734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:32909 (local address 127.0.0.1:51074)
0809 04:50:18.072021 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:18.072045 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:18.073255 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:18.074363 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:18.074368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:18.074393 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:18.074996 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:18.075009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.077037 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.077042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:18.079071 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:18.079104 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.080897 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.080904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:18.081067 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.081100 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 04:50:18.081189 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 04:50:18.081330 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1151,"thread_start_us":988,"threads_started":1}
I20260809 04:50:18.081718 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.071022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51074 (local address 127.31.93.1:32909)
0809 04:50:18.073813 (+ 2791us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:18.073819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:18.073840 (+ 21us) server_negotiation.cc:407] Connection header received
0809 04:50:18.073904 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:18.073909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:18.074120 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:18.074556 (+ 436us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:18.075177 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.076894 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.079276 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.080669 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.080821 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.081031 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:18.081254 (+ 223us) server_negotiation.cc:299] Negotiation successful
0809 04:50:18.081410 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2645,"thread_start_us":2493,"threads_started":1}
I20260809 04:50:18.083602 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.072625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:42287 (local address 127.0.0.1:39266)
0809 04:50:18.075189 (+ 2564us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:18.075210 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:18.075321 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:18.076250 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:18.076255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:18.076277 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:18.076701 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:18.076714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.078550 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.078556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:18.081383 (+ 2827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:18.081397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.082912 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.082918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:18.083086 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.083117 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 04:50:18.083194 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 04:50:18.083313 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1738,"thread_start_us":1523,"threads_started":1}
I20260809 04:50:18.083360 7079 tablet_service.cc:1467] Processing CreateTablet for tablet 22f07e0295164eb7a5ea10863966f7cf (DEFAULT_TABLE table=test-table [id=78b793c6ca734a3ea2a5afa0e87f9dbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:18.084317 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.072292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39266 (local address 127.31.93.2:42287)
0809 04:50:18.075126 (+ 2834us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:18.075134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:18.075296 (+ 162us) server_negotiation.cc:407] Connection header received
0809 04:50:18.075450 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:18.075457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:18.075664 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:18.076010 (+ 346us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:18.076858 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.078390 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.081570 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.082732 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.082842 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.083015 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:18.083859 (+ 844us) server_negotiation.cc:299] Negotiation successful
0809 04:50:18.084021 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2671,"thread_start_us":2582,"threads_started":1}
I20260809 04:50:18.084805 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22f07e0295164eb7a5ea10863966f7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:18.085772 6655 tablet_service.cc:1467] Processing CreateTablet for tablet 22f07e0295164eb7a5ea10863966f7cf (DEFAULT_TABLE table=test-table [id=78b793c6ca734a3ea2a5afa0e87f9dbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:18.087193 6655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22f07e0295164eb7a5ea10863966f7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:18.087769 6865 tablet_service.cc:1467] Processing CreateTablet for tablet 22f07e0295164eb7a5ea10863966f7cf (DEFAULT_TABLE table=test-table [id=78b793c6ca734a3ea2a5afa0e87f9dbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:18.089039 6865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22f07e0295164eb7a5ea10863966f7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:18.095965 7208 tablet_bootstrap.cc:492] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320: Bootstrap starting.
I20260809 04:50:18.099298 7208 tablet_bootstrap.cc:654] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:18.100256 7208 log.cc:824] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:18.101135 7209 tablet_bootstrap.cc:492] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36: Bootstrap starting.
I20260809 04:50:18.102550 7211 tablet_bootstrap.cc:492] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e: Bootstrap starting.
I20260809 04:50:18.104784 7209 tablet_bootstrap.cc:654] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:18.105158 7211 tablet_bootstrap.cc:654] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:18.106061 7211 log.cc:824] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:18.106058 7209 log.cc:824] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:18.108928 7211 tablet_bootstrap.cc:492] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e: No bootstrap required, opened a new log
I20260809 04:50:18.109205 7211 ts_tablet_manager.cc:1397] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260809 04:50:18.109313 7208 tablet_bootstrap.cc:492] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320: No bootstrap required, opened a new log
I20260809 04:50:18.109553 7208 ts_tablet_manager.cc:1397] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260809 04:50:18.109911 7209 tablet_bootstrap.cc:492] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36: No bootstrap required, opened a new log
I20260809 04:50:18.110176 7209 ts_tablet_manager.cc:1397] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 04:50:18.116204 7208 raft_consensus.cc:348] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1013419418b0403b85312e4e2c664320" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 41199 } } peers { permanent_uuid: "b265d37162724b7cb471071d71d88e36" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 32909 } } peers { permanent_uuid: "08381773767546648fe66e52759ecf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 42287 } }
I20260809 04:50:18.116534 7208 raft_consensus.cc:374] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:18.116667 7208 raft_consensus.cc:729] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1013419418b0403b85312e4e2c664320, State: Initialized, Role: FOLLOWER
I20260809 04:50:18.117393 7208 consensus_queue.cc:260] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1013419418b0403b85312e4e2c664320" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 41199 } } peers { permanent_uuid: "b265d37162724b7cb471071d71d88e36" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 32909 } } peers { permanent_uuid: "08381773767546648fe66e52759ecf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 42287 } }
I20260809 04:50:18.119273 7211 raft_consensus.cc:348] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1013419418b0403b85312e4e2c664320" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 41199 } } peers { permanent_uuid: "b265d37162724b7cb471071d71d88e36" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 32909 } } peers { permanent_uuid: "08381773767546648fe66e52759ecf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 42287 } }
I20260809 04:50:18.119704 7211 raft_consensus.cc:374] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:18.119825 7211 raft_consensus.cc:729] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08381773767546648fe66e52759ecf4e, State: Initialized, Role: FOLLOWER
I20260809 04:50:18.120110 7209 raft_consensus.cc:348] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1013419418b0403b85312e4e2c664320" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 41199 } } peers { permanent_uuid: "b265d37162724b7cb471071d71d88e36" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 32909 } } peers { permanent_uuid: "08381773767546648fe66e52759ecf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 42287 } }
I20260809 04:50:18.120527 7209 raft_consensus.cc:374] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:18.120643 7209 raft_consensus.cc:729] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b265d37162724b7cb471071d71d88e36, State: Initialized, Role: FOLLOWER
I20260809 04:50:18.120527 7211 consensus_queue.cc:260] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1013419418b0403b85312e4e2c664320" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 41199 } } peers { permanent_uuid: "b265d37162724b7cb471071d71d88e36" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 32909 } } peers { permanent_uuid: "08381773767546648fe66e52759ecf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 42287 } }
I20260809 04:50:18.121255 7209 consensus_queue.cc:260] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1013419418b0403b85312e4e2c664320" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 41199 } } peers { permanent_uuid: "b265d37162724b7cb471071d71d88e36" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 32909 } } peers { permanent_uuid: "08381773767546648fe66e52759ecf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 42287 } }
I20260809 04:50:18.121984 7175 heartbeater.cc:503] Master 127.31.93.62:33249 was elected leader, sending a full tablet report...
I20260809 04:50:18.122722 7208 ts_tablet_manager.cc:1428] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260809 04:50:18.123907 6963 heartbeater.cc:503] Master 127.31.93.62:33249 was elected leader, sending a full tablet report...
I20260809 04:50:18.125349 7209 ts_tablet_manager.cc:1428] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36: Time spent starting tablet: real 0.015s user 0.005s sys 0.010s
I20260809 04:50:18.126921 7211 ts_tablet_manager.cc:1428] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e: Time spent starting tablet: real 0.018s user 0.006s sys 0.011s
I20260809 04:50:18.202821 7220 raft_consensus.cc:482] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:50:18.203104 7220 raft_consensus.cc:504] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1013419418b0403b85312e4e2c664320" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 41199 } } peers { permanent_uuid: "b265d37162724b7cb471071d71d88e36" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 32909 } } peers { permanent_uuid: "08381773767546648fe66e52759ecf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 42287 } }
I20260809 04:50:18.204742 7220 leader_election.cc:288] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1013419418b0403b85312e4e2c664320 (127.31.93.3:41199), 08381773767546648fe66e52759ecf4e (127.31.93.2:42287)
I20260809 04:50:18.211486 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.205155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:37811 (local address 127.31.93.3:41199)
0809 04:50:18.205345 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:18.205349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:18.205736 (+ 387us) server_negotiation.cc:407] Connection header received
0809 04:50:18.206011 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:18.206016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:18.206095 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:18.206193 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:18.206968 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.207865 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.209682 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.210542 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.210638 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.210715 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:18.211092 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 04:50:18.211240 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 04:50:18.211517 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.206069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:49631 (local address 127.31.93.2:42287)
0809 04:50:18.206280 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:18.206286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:18.206520 (+ 234us) server_negotiation.cc:407] Connection header received
0809 04:50:18.206691 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:18.206694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:18.206749 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:18.206818 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:18.207352 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.208071 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.209822 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.210661 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.210794 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.210882 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:18.211161 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 04:50:18.211282 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 04:50:18.211608 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.205815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:42287 (local address 127.31.93.1:49631)
0809 04:50:18.206423 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:18.206444 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:18.206581 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:18.206854 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:18.206857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:18.206882 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:18.207206 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:18.207216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.208217 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.208221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:18.209665 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:18.209681 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.210876 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.210881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:18.211003 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.211031 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 04:50:18.211123 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 04:50:18.211241 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":301,"threads_started":1}
I20260809 04:50:18.211848 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.205010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:41199 (local address 127.31.93.1:37811)
0809 04:50:18.205645 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:18.205663 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:18.205798 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:18.206366 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:18.206370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:18.206398 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:18.206824 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:18.206836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.207989 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.207994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:18.209516 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:18.209529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.210747 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.210754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:18.210938 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.210966 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 04:50:18.211037 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 04:50:18.211149 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"spinlock_wait_cycles":45824,"thread_start_us":313,"threads_started":1}
I20260809 04:50:18.212638 6896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f07e0295164eb7a5ea10863966f7cf" candidate_uuid: "b265d37162724b7cb471071d71d88e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08381773767546648fe66e52759ecf4e" is_pre_election: true
I20260809 04:50:18.212688 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f07e0295164eb7a5ea10863966f7cf" candidate_uuid: "b265d37162724b7cb471071d71d88e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1013419418b0403b85312e4e2c664320" is_pre_election: true
I20260809 04:50:18.213090 6896 raft_consensus.cc:2454] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b265d37162724b7cb471071d71d88e36 in term 0.
I20260809 04:50:18.213119 7114 raft_consensus.cc:2454] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b265d37162724b7cb471071d71d88e36 in term 0.
I20260809 04:50:18.213676 6590 leader_election.cc:302] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1013419418b0403b85312e4e2c664320, b265d37162724b7cb471071d71d88e36; no voters:
I20260809 04:50:18.214013 7220 raft_consensus.cc:2781] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:50:18.214138 7220 raft_consensus.cc:482] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:50:18.214238 7220 raft_consensus.cc:3037] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:18.216374 7220 raft_consensus.cc:504] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1013419418b0403b85312e4e2c664320" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 41199 } } peers { permanent_uuid: "b265d37162724b7cb471071d71d88e36" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 32909 } } peers { permanent_uuid: "08381773767546648fe66e52759ecf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 42287 } }
I20260809 04:50:18.216915 7220 leader_election.cc:288] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [CANDIDATE]: Term 1 election: Requested vote from peers 1013419418b0403b85312e4e2c664320 (127.31.93.3:41199), 08381773767546648fe66e52759ecf4e (127.31.93.2:42287)
I20260809 04:50:18.217337 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f07e0295164eb7a5ea10863966f7cf" candidate_uuid: "b265d37162724b7cb471071d71d88e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1013419418b0403b85312e4e2c664320"
I20260809 04:50:18.217377 6896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22f07e0295164eb7a5ea10863966f7cf" candidate_uuid: "b265d37162724b7cb471071d71d88e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08381773767546648fe66e52759ecf4e"
I20260809 04:50:18.217504 7114 raft_consensus.cc:3037] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:18.217527 6896 raft_consensus.cc:3037] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:18.219664 7114 raft_consensus.cc:2454] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b265d37162724b7cb471071d71d88e36 in term 1.
I20260809 04:50:18.219704 6896 raft_consensus.cc:2454] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b265d37162724b7cb471071d71d88e36 in term 1.
I20260809 04:50:18.220009 6590 leader_election.cc:302] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1013419418b0403b85312e4e2c664320, b265d37162724b7cb471071d71d88e36; no voters:
I20260809 04:50:18.220266 7220 raft_consensus.cc:2781] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:18.220669 7220 raft_consensus.cc:686] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [term 1 LEADER]: Becoming Leader. State: Replica: b265d37162724b7cb471071d71d88e36, State: Running, Role: LEADER
I20260809 04:50:18.221165 7220 consensus_queue.cc:237] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1013419418b0403b85312e4e2c664320" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 41199 } } peers { permanent_uuid: "b265d37162724b7cb471071d71d88e36" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 32909 } } peers { permanent_uuid: "08381773767546648fe66e52759ecf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 42287 } }
I20260809 04:50:18.225538 6471 catalog_manager.cc:5495] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 reported cstate change: term changed from 0 to 1, leader changed from <none> to b265d37162724b7cb471071d71d88e36 (127.31.93.1). New cstate: current_term: 1 leader_uuid: "b265d37162724b7cb471071d71d88e36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1013419418b0403b85312e4e2c664320" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 41199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b265d37162724b7cb471071d71d88e36" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08381773767546648fe66e52759ecf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 42287 } health_report { overall_health: UNKNOWN } } }
W20260809 04:50:18.257581 7176 tablet.cc:2369] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:50:18.270241 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.265586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.31.93.1:32909)
0809 04:50:18.265757 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:18.265761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:18.265834 (+ 73us) server_negotiation.cc:407] Connection header received
0809 04:50:18.266009 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:18.266013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:18.266103 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:18.266229 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:50:18.266590 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.267302 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.268524 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.268906 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.268991 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.269374 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:50:18.269838 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:50:18.269942 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:18.269980 (+ 38us) server_negotiation.cc:299] Negotiation successful
0809 04:50:18.270042 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:50:18.277846 6896 raft_consensus.cc:1261] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e [term 1 FOLLOWER]: Refusing update from remote peer b265d37162724b7cb471071d71d88e36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:18.277892 7114 raft_consensus.cc:1261] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320 [term 1 FOLLOWER]: Refusing update from remote peer b265d37162724b7cb471071d71d88e36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:18.278756 7224 consensus_queue.cc:1035] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1013419418b0403b85312e4e2c664320" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:18.279096 7220 consensus_queue.cc:1035] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08381773767546648fe66e52759ecf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 42287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 04:50:18.287320 6964 tablet.cc:2369] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:50:18.287417 7233 mvcc.cc:205] Tried to move back new op lower bound from 7316484170857308160 to 7316484170638082048. Current Snapshot: MvccSnapshot[applied={T|T < 7316484170857308160}]
I20260809 04:50:18.290597 7234 mvcc.cc:205] Tried to move back new op lower bound from 7316484170857308160 to 7316484170638082048. Current Snapshot: MvccSnapshot[applied={T|T < 7316484170857308160}]
I20260809 04:50:18.293319 7235 mvcc.cc:205] Tried to move back new op lower bound from 7316484170857308160 to 7316484170638082048. Current Snapshot: MvccSnapshot[applied={T|T < 7316484170857308160}]
I20260809 04:50:18.303776 6476 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:18.304013 6476 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:18.307149 6476 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc3562bda7a04743bb2d1e50c61cf786: Sending DeleteTablet for 3 replicas of tablet 22f07e0295164eb7a5ea10863966f7cf
I20260809 04:50:18.308139 7079 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f07e0295164eb7a5ea10863966f7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:18 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:51592
I20260809 04:50:18.308308 6655 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f07e0295164eb7a5ea10863966f7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:18 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:51074
I20260809 04:50:18.308939 6865 tablet_service.cc:1514] Processing DeleteTablet for tablet 22f07e0295164eb7a5ea10863966f7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:18 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:39266
I20260809 04:50:18.309705 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 6565
I20260809 04:50:18.323455 7251 tablet_replica.cc:330] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e: stopping tablet replica
I20260809 04:50:18.323948 7251 raft_consensus.cc:2229] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:18.324484 7251 raft_consensus.cc:2258] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:18.326910 7249 tablet_replica.cc:330] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36: stopping tablet replica
I20260809 04:50:18.327428 7249 raft_consensus.cc:2229] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:50:18.328202 7249 raft_consensus.cc:2258] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:18.330464 7251 ts_tablet_manager.cc:1905] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:18.330781 7247 tablet_replica.cc:330] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320: stopping tablet replica
I20260809 04:50:18.331176 7247 raft_consensus.cc:2229] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:18.331642 7247 raft_consensus.cc:2258] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:18.333202 7249 ts_tablet_manager.cc:1905] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:18.333914 7247 ts_tablet_manager.cc:1905] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 04:50:18 dist-test-slave-4tdj krb5kdc[6410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:50:18.337497 7251 ts_tablet_manager.cc:1918] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:18.337677 7251 log.cc:1196] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/22f07e0295164eb7a5ea10863966f7cf
I20260809 04:50:18.338225 7251 ts_tablet_manager.cc:1939] T 22f07e0295164eb7a5ea10863966f7cf P 08381773767546648fe66e52759ecf4e: Deleting consensus metadata
I20260809 04:50:18.340142 6451 catalog_manager.cc:4841] TS 08381773767546648fe66e52759ecf4e (127.31.93.2:42287): tablet 22f07e0295164eb7a5ea10863966f7cf (table test-table [id=78b793c6ca734a3ea2a5afa0e87f9dbe]) successfully deleted
I20260809 04:50:18.341216 7247 ts_tablet_manager.cc:1918] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:18.341387 7247 log.cc:1196] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/22f07e0295164eb7a5ea10863966f7cf
I20260809 04:50:18.341910 7247 ts_tablet_manager.cc:1939] T 22f07e0295164eb7a5ea10863966f7cf P 1013419418b0403b85312e4e2c664320: Deleting consensus metadata
I20260809 04:50:18.343586 6451 catalog_manager.cc:4841] TS 1013419418b0403b85312e4e2c664320 (127.31.93.3:41199): tablet 22f07e0295164eb7a5ea10863966f7cf (table test-table [id=78b793c6ca734a3ea2a5afa0e87f9dbe]) successfully deleted
I20260809 04:50:18.345688 7249 ts_tablet_manager.cc:1918] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:18.345856 7249 log.cc:1196] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/22f07e0295164eb7a5ea10863966f7cf
I20260809 04:50:18.346336 7249 ts_tablet_manager.cc:1939] T 22f07e0295164eb7a5ea10863966f7cf P b265d37162724b7cb471071d71d88e36: Deleting consensus metadata
I20260809 04:50:18.347834 6451 catalog_manager.cc:4841] TS b265d37162724b7cb471071d71d88e36 (127.31.93.1:32909): tablet 22f07e0295164eb7a5ea10863966f7cf (table test-table [id=78b793c6ca734a3ea2a5afa0e87f9dbe]) successfully deleted
I20260809 04:50:18.348949 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.311063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.31.93.1:32909)
0809 04:50:18.311237 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:18.311245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:18.320499 (+ 9254us) server_negotiation.cc:407] Connection header received
0809 04:50:18.320955 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:18.320961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:18.321047 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:18.321124 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:18.324918 (+ 3794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.326261 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.328396 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.329149 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.329277 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.341602 (+ 12325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:18.341615 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:18.341631 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:18.341671 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:18.345805 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:18.346753 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:18.346762 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:18.346765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:18.346963 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:18.347416 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:18.347421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:18.347424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:18.347866 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:18.348161 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:18.348497 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 04:50:18.348685 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 04:50:18.349531 6741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:18.446401 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 6565
I20260809 04:50:18.469702 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 6762
Aug 09 04:50:18 dist-test-slave-4tdj krb5kdc[6410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:50:18.483155 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.470521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39282 (local address 127.31.93.2:42287)
0809 04:50:18.470684 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:18.470688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:18.470706 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:50:18.470859 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:18.470863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:18.470953 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:18.471041 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:18.472167 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.472896 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.473993 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.474453 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.474546 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.478269 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:18.478282 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:18.478325 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:18.478365 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:18.480976 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:18.481513 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:18.481519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:18.481522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:18.481647 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:18.481908 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:18.481911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:18.481913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:18.482240 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:18.482530 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:18.482657 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 04:50:18.482810 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 04:50:18.483816 6955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:18.570017 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 6762
I20260809 04:50:18.593012 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 6970
Aug 09 04:50:18 dist-test-slave-4tdj krb5kdc[6410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:50:18.606349 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:18.593716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.31.93.3:41199)
0809 04:50:18.593869 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:18.593874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:18.593894 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:50:18.594050 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:18.594054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:18.594132 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:18.594213 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:18.595414 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.596134 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.597271 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:18.597634 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:18.597717 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:18.601642 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:18.601658 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:18.601674 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:18.601710 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:18.604191 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:18.604737 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:18.604743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:18.604746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:18.604867 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:18.605142 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:18.605145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:18.605147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:18.605488 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:18.605704 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:18.605909 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 04:50:18.606063 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 04:50:18.606966 7168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:18.693929 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 6970
I20260809 04:50:18.713223 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 6429
I20260809 04:50:18.714447 6519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:18.820408 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 6429
2026-08-09T04:50:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4082 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-09T04:50:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:50:18Z Disabled control of system clock
I20260809 04:50:18.892719 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:40509
--webserver_interface=127.31.93.62
--webserver_port=0
--builtin_ntp_servers=127.31.93.20:46565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:40509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 04:50:19.204317 7270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:19.204567 7270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:19.204659 7270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:19.211514 7270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:19.211606 7270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:19.211649 7270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:19.211702 7270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:19.211745 7270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:50:19.220327 7270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:46565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:40509
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:40509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:19.221459 7270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:19.223122 7270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:19.234050 7277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:19.234028 7284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:19.234052 7278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:19.234777 7270 server_base.cc:1034] running on GCE node
I20260809 04:50:19.235467 7270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:19.236970 7270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:19.238230 7270 hybrid_clock.cc:648] HybridClock initialized: now 1786251019238180 us; error 41 us; skew 500 ppm
I20260809 04:50:19.238569 7270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:50:19.240554 7270 webserver.cc:460] Webserver started at http://127.31.93.62:44895/ using document root <none> and password file <none>
I20260809 04:50:19.241178 7270 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:19.241271 7270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:19.241573 7270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:19.244282 7270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "4328ba20bbf040148cc46c4f0840e262"
format_stamp: "Formatted at 2026-08-09 04:50:19 on dist-test-slave-4tdj"
server_key: "1e1f876d5be46fa2185d60c1712726cb"
server_key_iv: "ea9ead01d4a6757c2f148bbc0551fe48"
server_key_version: "encryptionkey@0"
I20260809 04:50:19.244997 7270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "4328ba20bbf040148cc46c4f0840e262"
format_stamp: "Formatted at 2026-08-09 04:50:19 on dist-test-slave-4tdj"
server_key: "1e1f876d5be46fa2185d60c1712726cb"
server_key_iv: "ea9ead01d4a6757c2f148bbc0551fe48"
server_key_version: "encryptionkey@0"
I20260809 04:50:19.249727 7270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260809 04:50:19.252867 7297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:19.254060 7270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:50:19.254216 7270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "4328ba20bbf040148cc46c4f0840e262"
format_stamp: "Formatted at 2026-08-09 04:50:19 on dist-test-slave-4tdj"
server_key: "1e1f876d5be46fa2185d60c1712726cb"
server_key_iv: "ea9ead01d4a6757c2f148bbc0551fe48"
server_key_version: "encryptionkey@0"
I20260809 04:50:19.254451 7270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:19.259528 7270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:19.260211 7270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:19.260479 7270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:19.276862 7270 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:40509
I20260809 04:50:19.276895 7379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:40509 every 8 connection(s)
I20260809 04:50:19.278013 7270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:50:19.281421 7380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:19.287753 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 7270
I20260809 04:50:19.287961 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:50:19.288753 32116 external_mini_cluster.cc:1428] Setting key 3435ad4771ce458832774aeb5b0d0ce1
I20260809 04:50:19.292024 7380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262: Bootstrap starting.
I20260809 04:50:19.295185 7380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:19.296274 7380 log.cc:824] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:19.296519 7383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:19.290739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32986 (local address 127.31.93.62:40509)
0809 04:50:19.291629 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:19.291639 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:19.291678 (+ 39us) server_negotiation.cc:407] Connection header received
0809 04:50:19.293276 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:19.293300 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:19.293669 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:19.293956 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:19.294990 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:19.294995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:19.295006 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:19.295020 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:19.295071 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:19.295100 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:19.295232 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:19.295429 (+ 197us) server_negotiation.cc:299] Negotiation successful
0809 04:50:19.295686 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":305,"threads_started":1}
I20260809 04:50:19.298938 7380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262: No bootstrap required, opened a new log
I20260809 04:50:19.305495 7380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4328ba20bbf040148cc46c4f0840e262" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 40509 } }
I20260809 04:50:19.305804 7380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:19.305887 7380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4328ba20bbf040148cc46c4f0840e262, State: Initialized, Role: FOLLOWER
I20260809 04:50:19.306439 7380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4328ba20bbf040148cc46c4f0840e262" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 40509 } }
I20260809 04:50:19.306661 7380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:50:19.306782 7380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:50:19.306931 7380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:19.308853 7380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4328ba20bbf040148cc46c4f0840e262" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 40509 } }
I20260809 04:50:19.309294 7380 leader_election.cc:302] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4328ba20bbf040148cc46c4f0840e262; no voters:
I20260809 04:50:19.309883 7380 leader_election.cc:288] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:50:19.309980 7387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:19.310670 7387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [term 1 LEADER]: Becoming Leader. State: Replica: 4328ba20bbf040148cc46c4f0840e262, State: Running, Role: LEADER
I20260809 04:50:19.311234 7387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4328ba20bbf040148cc46c4f0840e262" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 40509 } }
I20260809 04:50:19.311770 7380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:19.313719 7389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4328ba20bbf040148cc46c4f0840e262. Latest consensus state: current_term: 1 leader_uuid: "4328ba20bbf040148cc46c4f0840e262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4328ba20bbf040148cc46c4f0840e262" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 40509 } } }
I20260809 04:50:19.313864 7388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4328ba20bbf040148cc46c4f0840e262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4328ba20bbf040148cc46c4f0840e262" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 40509 } } }
I20260809 04:50:19.314091 7389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:19.314160 7388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262 [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:19.315879 7400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:50:19.320685 7400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:50:19.328814 7400 catalog_manager.cc:1347] Generated new cluster ID: f6e689d5dcff4415af8ee8c568feefc1
I20260809 04:50:19.328918 7400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:50:19.346732 7400 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:50:19.347157 7400 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:50:19.353904 7400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262: Generated new TSK 0
I20260809 04:50:19.354979 7400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:50:19.366689 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:40509
--builtin_ntp_servers=127.31.93.20:46565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 04:50:19.680330 7419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:19.680572 7419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:19.680689 7419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:19.687695 7419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:19.687780 7419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:19.687922 7419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:50:19.696501 7419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:46565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:40509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:19.697656 7419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:19.699399 7419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:19.723482 7427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:19.723879 7430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:19.724035 7426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:19.725544 7419 server_base.cc:1034] running on GCE node
I20260809 04:50:19.726145 7419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:19.727072 7419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:19.728274 7419 hybrid_clock.cc:648] HybridClock initialized: now 1786251019728221 us; error 45 us; skew 500 ppm
I20260809 04:50:19.728569 7419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:50:19.730834 7419 webserver.cc:460] Webserver started at http://127.31.93.1:38739/ using document root <none> and password file <none>
I20260809 04:50:19.731477 7419 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:19.731583 7419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:19.731894 7419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:19.734638 7419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "fed195921b994cf589956abebb7bf697"
format_stamp: "Formatted at 2026-08-09 04:50:19 on dist-test-slave-4tdj"
server_key: "fce8f52c59fe787aa47b263f075e0dd9"
server_key_iv: "099d9fb662616d35cf48684d813bde43"
server_key_version: "encryptionkey@0"
I20260809 04:50:19.735368 7419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "fed195921b994cf589956abebb7bf697"
format_stamp: "Formatted at 2026-08-09 04:50:19 on dist-test-slave-4tdj"
server_key: "fce8f52c59fe787aa47b263f075e0dd9"
server_key_iv: "099d9fb662616d35cf48684d813bde43"
server_key_version: "encryptionkey@0"
I20260809 04:50:19.740368 7419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 04:50:19.743772 7449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:19.745076 7419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:50:19.745239 7419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "fed195921b994cf589956abebb7bf697"
format_stamp: "Formatted at 2026-08-09 04:50:19 on dist-test-slave-4tdj"
server_key: "fce8f52c59fe787aa47b263f075e0dd9"
server_key_iv: "099d9fb662616d35cf48684d813bde43"
server_key_version: "encryptionkey@0"
I20260809 04:50:19.745435 7419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:19.750674 7419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:19.751389 7419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:19.751643 7419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:19.752732 7419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:19.754086 7419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:19.754184 7419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:19.754283 7419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:19.754403 7419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260809 04:50:19.795346 7419 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:40735
I20260809 04:50:19.795357 7606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:40735 every 8 connection(s)
I20260809 04:50:19.796738 7419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:50:19.801916 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 7419
I20260809 04:50:19.802091 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:50:19.802919 32116 external_mini_cluster.cc:1428] Setting key d6c2df0673d452508e510c152d7427f3
I20260809 04:50:19.804966 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:19.799117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:52367 (local address 127.31.93.62:40509)
0809 04:50:19.799491 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:19.799496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:19.800711 (+ 1215us) server_negotiation.cc:407] Connection header received
0809 04:50:19.802707 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:19.802711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:19.802766 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:19.802858 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:19.804248 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:19.804252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:19.804255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:19.804271 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:19.804311 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:19.804322 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:19.804426 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:19.804648 (+ 222us) server_negotiation.cc:299] Negotiation successful
0809 04:50:19.804713 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":181,"threads_started":1}
I20260809 04:50:19.805917 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:40509
--builtin_ntp_servers=127.31.93.20:46565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 04:50:19.806033 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:19.799695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:40509 (local address 127.31.93.1:52367)
0809 04:50:19.800562 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:19.800604 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:19.802425 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:19.803119 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:19.803128 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:19.803237 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:19.803955 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:50:19.803959 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:19.804007 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:50:19.804092 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:19.804407 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:19.804417 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:19.804424 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 04:50:19.804691 (+ 267us) client_negotiation.cc:240] Negotiation successful
0809 04:50:19.804903 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":261,"threads_started":1}
I20260809 04:50:19.807384 7608 heartbeater.cc:347] Connected to a master server at 127.31.93.62:40509
I20260809 04:50:19.807673 7608 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:19.808274 7608 heartbeater.cc:511] Master 127.31.93.62:40509 requested a full tablet report, sending...
I20260809 04:50:19.810230 7325 ts_manager.cc:194] Registered new tserver with Master: fed195921b994cf589956abebb7bf697 (127.31.93.1:40735)
W20260809 04:50:20.126698 7615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:20.127008 7615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:20.127153 7615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:20.134414 7615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:20.134539 7615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:20.134704 7615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:50:20.143765 7615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:46565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:40509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:20.145093 7615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:20.147025 7615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:20.160024 7625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:20.160168 7627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:20.160203 7623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:20.161003 7615 server_base.cc:1034] running on GCE node
I20260809 04:50:20.161628 7615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:20.162592 7615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:20.163805 7615 hybrid_clock.cc:648] HybridClock initialized: now 1786251020163755 us; error 37 us; skew 500 ppm
I20260809 04:50:20.164126 7615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:50:20.166394 7615 webserver.cc:460] Webserver started at http://127.31.93.2:34051/ using document root <none> and password file <none>
I20260809 04:50:20.167064 7615 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:20.167176 7615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:20.167517 7615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:20.170357 7615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "eca8766fa25740338dd1e6fad45ebf4e"
format_stamp: "Formatted at 2026-08-09 04:50:20 on dist-test-slave-4tdj"
server_key: "a3f0945c93a58d8a99cf56adc081494e"
server_key_iv: "c9f80418719aad7526dbd48d6c67516d"
server_key_version: "encryptionkey@0"
I20260809 04:50:20.171159 7615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "eca8766fa25740338dd1e6fad45ebf4e"
format_stamp: "Formatted at 2026-08-09 04:50:20 on dist-test-slave-4tdj"
server_key: "a3f0945c93a58d8a99cf56adc081494e"
server_key_iv: "c9f80418719aad7526dbd48d6c67516d"
server_key_version: "encryptionkey@0"
I20260809 04:50:20.176172 7615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 04:50:20.179535 7636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:20.180745 7615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:50:20.180909 7615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "eca8766fa25740338dd1e6fad45ebf4e"
format_stamp: "Formatted at 2026-08-09 04:50:20 on dist-test-slave-4tdj"
server_key: "a3f0945c93a58d8a99cf56adc081494e"
server_key_iv: "c9f80418719aad7526dbd48d6c67516d"
server_key_version: "encryptionkey@0"
I20260809 04:50:20.181116 7615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:20.186704 7615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:20.187492 7615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:20.187793 7615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:20.188736 7615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:20.190127 7615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:20.190232 7615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:20.190377 7615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:20.190462 7615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:20.221150 7615 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:43423
I20260809 04:50:20.221200 7815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:43423 every 8 connection(s)
I20260809 04:50:20.222553 7615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:50:20.225311 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 7615
I20260809 04:50:20.225461 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:50:20.225899 32116 external_mini_cluster.cc:1428] Setting key 89dabe76b98fa7a0b3e57c87eaab6364
I20260809 04:50:20.228654 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:40509
--builtin_ntp_servers=127.31.93.20:46565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 04:50:20.231268 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.225023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:41245 (local address 127.31.93.62:40509)
0809 04:50:20.225262 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:20.225269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:20.226839 (+ 1570us) server_negotiation.cc:407] Connection header received
0809 04:50:20.228661 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:20.228666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:20.228733 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:20.228841 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:20.230527 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:20.230532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:20.230537 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:20.230558 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:20.230596 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:20.230610 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:20.230742 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:20.230920 (+ 178us) server_negotiation.cc:299] Negotiation successful
0809 04:50:20.230986 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 04:50:20.232173 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.225604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:40509 (local address 127.31.93.2:41245)
0809 04:50:20.226688 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:20.226733 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:20.228396 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:20.229226 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:20.229239 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:20.229357 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:20.230066 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:50:20.230070 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:20.230149 (+ 79us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:50:20.230263 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:20.230706 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:20.230719 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:20.230727 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 04:50:20.230984 (+ 257us) client_negotiation.cc:240] Negotiation successful
0809 04:50:20.231242 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":319,"threads_started":1}
I20260809 04:50:20.233423 7817 heartbeater.cc:347] Connected to a master server at 127.31.93.62:40509
I20260809 04:50:20.233700 7817 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:20.234234 7817 heartbeater.cc:511] Master 127.31.93.62:40509 requested a full tablet report, sending...
I20260809 04:50:20.235576 7325 ts_manager.cc:194] Registered new tserver with Master: eca8766fa25740338dd1e6fad45ebf4e (127.31.93.2:43423)
W20260809 04:50:20.542827 7820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:20.543076 7820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:20.543169 7820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:20.550077 7820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:20.550174 7820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:20.550287 7820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:50:20.558956 7820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:46565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:40509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:20.560063 7820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:20.561757 7820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:20.574606 7833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:20.574689 7842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:20.574591 7834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:20.575768 7820 server_base.cc:1034] running on GCE node
I20260809 04:50:20.576395 7820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:20.577312 7820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:20.578482 7820 hybrid_clock.cc:648] HybridClock initialized: now 1786251020578427 us; error 51 us; skew 500 ppm
I20260809 04:50:20.578758 7820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:50:20.580765 7820 webserver.cc:460] Webserver started at http://127.31.93.3:34205/ using document root <none> and password file <none>
I20260809 04:50:20.581348 7820 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:20.581432 7820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:20.581710 7820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:20.584365 7820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "e4321eae55644c45a0f06238b1b6db9f"
format_stamp: "Formatted at 2026-08-09 04:50:20 on dist-test-slave-4tdj"
server_key: "98d8cc9873c5b026e26e78d7df060b5d"
server_key_iv: "17917e9cdde75c7690e61132830cb0b7"
server_key_version: "encryptionkey@0"
I20260809 04:50:20.585094 7820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "e4321eae55644c45a0f06238b1b6db9f"
format_stamp: "Formatted at 2026-08-09 04:50:20 on dist-test-slave-4tdj"
server_key: "98d8cc9873c5b026e26e78d7df060b5d"
server_key_iv: "17917e9cdde75c7690e61132830cb0b7"
server_key_version: "encryptionkey@0"
I20260809 04:50:20.589920 7820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 04:50:20.593171 7855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:20.594456 7820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:50:20.594587 7820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "e4321eae55644c45a0f06238b1b6db9f"
format_stamp: "Formatted at 2026-08-09 04:50:20 on dist-test-slave-4tdj"
server_key: "98d8cc9873c5b026e26e78d7df060b5d"
server_key_iv: "17917e9cdde75c7690e61132830cb0b7"
server_key_version: "encryptionkey@0"
I20260809 04:50:20.594780 7820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:20.600062 7820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:20.600773 7820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:20.601009 7820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:20.601881 7820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:20.603214 7820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:20.603299 7820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:20.603404 7820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:20.603475 7820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:20.632990 7820 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:36297
I20260809 04:50:20.633066 8063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:36297 every 8 connection(s)
I20260809 04:50:20.634238 7820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:50:20.639674 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 7820
I20260809 04:50:20.639853 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:50:20.640465 32116 external_mini_cluster.cc:1428] Setting key b2f2e6b259ef9a0cc84452fdf52c2177
I20260809 04:50:20.642598 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.636741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:45895 (local address 127.31.93.62:40509)
0809 04:50:20.636919 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:20.636923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:20.638268 (+ 1345us) server_negotiation.cc:407] Connection header received
0809 04:50:20.640293 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:20.640298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:20.640352 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:20.640435 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:20.641815 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:20.641821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:20.641825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:20.641841 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:20.641873 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:20.641884 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:20.641968 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:20.642227 (+ 259us) server_negotiation.cc:299] Negotiation successful
0809 04:50:20.642314 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 04:50:20.643746 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.637192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:40509 (local address 127.31.93.3:45895)
0809 04:50:20.638105 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:20.638151 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:20.639997 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:20.640704 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:20.640716 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:20.640830 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:20.641548 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:50:20.641550 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:20.641588 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:50:20.641673 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:20.641985 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:20.641995 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:20.642004 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 04:50:20.642333 (+ 329us) client_negotiation.cc:240] Negotiation successful
0809 04:50:20.642603 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":273,"threads_started":1}
I20260809 04:50:20.644976 8066 heartbeater.cc:347] Connected to a master server at 127.31.93.62:40509
I20260809 04:50:20.645243 8066 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:20.645783 8066 heartbeater.cc:511] Master 127.31.93.62:40509 requested a full tablet report, sending...
I20260809 04:50:20.647104 7325 ts_manager.cc:194] Registered new tserver with Master: e4321eae55644c45a0f06238b1b6db9f (127.31.93.3:36297)
I20260809 04:50:20.655756 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:50:20.662540 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.660271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.31.93.62:40509)
0809 04:50:20.660410 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:20.660414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:20.660921 (+ 507us) server_negotiation.cc:407] Connection header received
0809 04:50:20.661036 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:20.661040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:20.661094 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:20.661177 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:20.661952 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:20.661956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:20.661959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:20.661978 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:20.662010 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:20.662023 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:20.662120 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:20.662163 (+ 43us) server_negotiation.cc:299] Negotiation successful
0809 04:50:20.662208 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 04:50:20.666273 7325 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:50:20.673074 7325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:50:20.690945 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.684500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:36297 (local address 127.0.0.1:41450)
0809 04:50:20.685442 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:20.685489 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:20.685722 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:20.687633 (+ 1911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:20.687643 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:20.687745 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:20.688551 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:50:20.688555 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:20.688585 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:50:20.688679 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:20.690139 (+ 1460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:20.690153 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:20.690163 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 04:50:20.690357 (+ 194us) client_negotiation.cc:240] Negotiation successful
0809 04:50:20.690448 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":308,"threads_started":1}
I20260809 04:50:20.692359 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.684742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41450 (local address 127.31.93.3:36297)
0809 04:50:20.687048 (+ 2306us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:20.687058 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:20.687099 (+ 41us) server_negotiation.cc:407] Connection header received
0809 04:50:20.687174 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:20.687183 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:20.687413 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:20.688228 (+ 815us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:20.689868 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:20.689878 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:20.689893 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:20.689911 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:20.689965 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:20.689998 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:20.691741 (+ 1743us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:20.691823 (+ 82us) server_negotiation.cc:299] Negotiation successful
0809 04:50:20.691928 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2112,"thread_start_us":1930,"threads_started":1}
I20260809 04:50:20.698204 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.684995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:43423 (local address 127.0.0.1:42318)
0809 04:50:20.691406 (+ 6411us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:20.691426 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:20.691511 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:20.695830 (+ 4319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:20.695834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:20.695873 (+ 39us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:20.696398 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:50:20.696401 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:20.696417 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:50:20.696460 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:20.697779 (+ 1319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:20.697784 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:20.697788 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 04:50:20.697880 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 04:50:20.697932 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6302,"thread_start_us":10232,"threads_started":1}
I20260809 04:50:20.698488 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.685505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42318 (local address 127.31.93.2:43423)
0809 04:50:20.695258 (+ 9753us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:20.695268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:20.695304 (+ 36us) server_negotiation.cc:407] Connection header received
0809 04:50:20.695384 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:20.695393 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:20.695627 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:20.695848 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:20.697481 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:20.697491 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:20.697505 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:20.697536 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:20.697596 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:20.697620 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:20.697904 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:20.697993 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 04:50:20.698093 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9568,"thread_start_us":9413,"threads_started":1}
I20260809 04:50:20.699410 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.685673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:40735 (local address 127.0.0.1:49144)
0809 04:50:20.695534 (+ 9861us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:20.695557 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:20.695684 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:20.696216 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:20.696250 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:20.696272 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:20.697523 (+ 1251us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:50:20.697525 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:20.697537 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:50:20.697573 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:20.698082 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:20.698086 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:20.698091 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 04:50:20.698242 (+ 151us) client_negotiation.cc:240] Negotiation successful
0809 04:50:20.699016 (+ 774us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9718,"thread_start_us":9882,"threads_started":1}
I20260809 04:50:20.699608 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.686223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49144 (local address 127.31.93.1:40735)
0809 04:50:20.694768 (+ 8545us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:20.694779 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:20.695761 (+ 982us) server_negotiation.cc:407] Connection header received
0809 04:50:20.695836 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:20.695844 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:20.696054 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:20.696205 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:20.697725 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:20.697733 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:20.697743 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:20.697763 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:20.697826 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:20.697847 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:20.698031 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:20.698575 (+ 544us) server_negotiation.cc:299] Negotiation successful
0809 04:50:20.698681 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8375,"thread_start_us":8255,"threads_started":1}
I20260809 04:50:20.700340 7938 tablet_service.cc:1467] Processing CreateTablet for tablet 535d374e28194260912c5e8167d4922e (DEFAULT_TABLE table=test-table [id=fdbbad1edb75490ea43d3322c3e3c517]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:20.702163 7715 tablet_service.cc:1467] Processing CreateTablet for tablet 535d374e28194260912c5e8167d4922e (DEFAULT_TABLE table=test-table [id=fdbbad1edb75490ea43d3322c3e3c517]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:20.703655 7715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 535d374e28194260912c5e8167d4922e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:20.704576 7519 tablet_service.cc:1467] Processing CreateTablet for tablet 535d374e28194260912c5e8167d4922e (DEFAULT_TABLE table=test-table [id=fdbbad1edb75490ea43d3322c3e3c517]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:20.705265 7938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 535d374e28194260912c5e8167d4922e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:20.705952 7519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 535d374e28194260912c5e8167d4922e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:20.716403 8094 tablet_bootstrap.cc:492] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f: Bootstrap starting.
I20260809 04:50:20.718773 8095 tablet_bootstrap.cc:492] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e: Bootstrap starting.
I20260809 04:50:20.719988 8094 tablet_bootstrap.cc:654] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:20.720501 8096 tablet_bootstrap.cc:492] T 535d374e28194260912c5e8167d4922e P fed195921b994cf589956abebb7bf697: Bootstrap starting.
I20260809 04:50:20.721304 8094 log.cc:824] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:20.722285 8095 tablet_bootstrap.cc:654] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:20.723460 8095 log.cc:824] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:20.724126 8096 tablet_bootstrap.cc:654] T 535d374e28194260912c5e8167d4922e P fed195921b994cf589956abebb7bf697: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:20.724885 8094 tablet_bootstrap.cc:492] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f: No bootstrap required, opened a new log
I20260809 04:50:20.725185 8094 ts_tablet_manager.cc:1397] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 04:50:20.725484 8096 log.cc:824] T 535d374e28194260912c5e8167d4922e P fed195921b994cf589956abebb7bf697: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:20.726034 8095 tablet_bootstrap.cc:492] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e: No bootstrap required, opened a new log
I20260809 04:50:20.726378 8095 ts_tablet_manager.cc:1397] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260809 04:50:20.728464 8096 tablet_bootstrap.cc:492] T 535d374e28194260912c5e8167d4922e P fed195921b994cf589956abebb7bf697: No bootstrap required, opened a new log
I20260809 04:50:20.728730 8096 ts_tablet_manager.cc:1397] T 535d374e28194260912c5e8167d4922e P fed195921b994cf589956abebb7bf697: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 04:50:20.733062 8095 raft_consensus.cc:348] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4321eae55644c45a0f06238b1b6db9f" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36297 } } peers { permanent_uuid: "fed195921b994cf589956abebb7bf697" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 40735 } } peers { permanent_uuid: "eca8766fa25740338dd1e6fad45ebf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43423 } }
I20260809 04:50:20.733500 8095 raft_consensus.cc:374] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:20.733634 8095 raft_consensus.cc:729] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca8766fa25740338dd1e6fad45ebf4e, State: Initialized, Role: FOLLOWER
I20260809 04:50:20.734390 8095 consensus_queue.cc:260] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4321eae55644c45a0f06238b1b6db9f" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36297 } } peers { permanent_uuid: "fed195921b994cf589956abebb7bf697" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 40735 } } peers { permanent_uuid: "eca8766fa25740338dd1e6fad45ebf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43423 } }
I20260809 04:50:20.735344 7817 heartbeater.cc:503] Master 127.31.93.62:40509 was elected leader, sending a full tablet report...
I20260809 04:50:20.735962 8095 ts_tablet_manager.cc:1428] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20260809 04:50:20.735733 8094 raft_consensus.cc:348] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4321eae55644c45a0f06238b1b6db9f" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36297 } } peers { permanent_uuid: "fed195921b994cf589956abebb7bf697" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 40735 } } peers { permanent_uuid: "eca8766fa25740338dd1e6fad45ebf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43423 } }
I20260809 04:50:20.736158 8094 raft_consensus.cc:374] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:20.736336 8094 raft_consensus.cc:729] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4321eae55644c45a0f06238b1b6db9f, State: Initialized, Role: FOLLOWER
I20260809 04:50:20.737061 8094 consensus_queue.cc:260] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4321eae55644c45a0f06238b1b6db9f" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36297 } } peers { permanent_uuid: "fed195921b994cf589956abebb7bf697" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 40735 } } peers { permanent_uuid: "eca8766fa25740338dd1e6fad45ebf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43423 } }
I20260809 04:50:20.738180 8066 heartbeater.cc:503] Master 127.31.93.62:40509 was elected leader, sending a full tablet report...
I20260809 04:50:20.738865 8094 ts_tablet_manager.cc:1428] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260809 04:50:20.739069 8096 raft_consensus.cc:348] T 535d374e28194260912c5e8167d4922e P fed195921b994cf589956abebb7bf697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4321eae55644c45a0f06238b1b6db9f" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36297 } } peers { permanent_uuid: "fed195921b994cf589956abebb7bf697" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 40735 } } peers { permanent_uuid: "eca8766fa25740338dd1e6fad45ebf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43423 } }
I20260809 04:50:20.739521 8096 raft_consensus.cc:374] T 535d374e28194260912c5e8167d4922e P fed195921b994cf589956abebb7bf697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:20.739677 8096 raft_consensus.cc:729] T 535d374e28194260912c5e8167d4922e P fed195921b994cf589956abebb7bf697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed195921b994cf589956abebb7bf697, State: Initialized, Role: FOLLOWER
I20260809 04:50:20.740267 8096 consensus_queue.cc:260] T 535d374e28194260912c5e8167d4922e P fed195921b994cf589956abebb7bf697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4321eae55644c45a0f06238b1b6db9f" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36297 } } peers { permanent_uuid: "fed195921b994cf589956abebb7bf697" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 40735 } } peers { permanent_uuid: "eca8766fa25740338dd1e6fad45ebf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43423 } }
I20260809 04:50:20.741380 7608 heartbeater.cc:503] Master 127.31.93.62:40509 was elected leader, sending a full tablet report...
I20260809 04:50:20.741959 8096 ts_tablet_manager.cc:1428] T 535d374e28194260912c5e8167d4922e P fed195921b994cf589956abebb7bf697: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260809 04:50:20.753103 8103 raft_consensus.cc:482] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:50:20.753340 8103 raft_consensus.cc:504] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4321eae55644c45a0f06238b1b6db9f" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36297 } } peers { permanent_uuid: "fed195921b994cf589956abebb7bf697" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 40735 } } peers { permanent_uuid: "eca8766fa25740338dd1e6fad45ebf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43423 } }
I20260809 04:50:20.754817 8103 leader_election.cc:288] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fed195921b994cf589956abebb7bf697 (127.31.93.1:40735), eca8766fa25740338dd1e6fad45ebf4e (127.31.93.2:43423)
I20260809 04:50:20.757221 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.755428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:33767 (local address 127.31.93.2:43423)
0809 04:50:20.755585 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:20.755590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:20.755604 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:50:20.755643 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:20.755647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:20.755689 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:20.755766 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:20.756631 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:20.756636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:20.756641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:20.756661 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:20.756696 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:20.756711 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:20.756832 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:20.756936 (+ 104us) server_negotiation.cc:299] Negotiation successful
0809 04:50:20.756984 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:50:20.757272 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.755094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:43423 (local address 127.31.93.3:33767)
0809 04:50:20.755353 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:20.755372 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:20.755460 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:20.755825 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:20.755833 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:20.755863 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:20.756408 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:50:20.756410 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:20.756428 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:50:20.756472 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:20.756813 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:20.756818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:20.756824 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 04:50:20.756943 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 04:50:20.756993 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260809 04:50:20.757966 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.755167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:35315 (local address 127.31.93.1:40735)
0809 04:50:20.755516 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:20.755523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:20.756041 (+ 518us) server_negotiation.cc:407] Connection header received
0809 04:50:20.756171 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:20.756174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:20.756216 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:20.756291 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:20.757173 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:20.757179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:20.757184 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:20.757202 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:20.757234 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:20.757249 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:20.757334 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:20.757670 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 04:50:20.757729 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260809 04:50:20.758033 8106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.755154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:40735 (local address 127.31.93.3:35315)
0809 04:50:20.755948 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:20.755962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:20.756065 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:20.756325 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:20.756330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:20.756352 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:20.756953 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:50:20.756956 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:20.756971 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:50:20.757010 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:20.757461 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:20.757466 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:20.757469 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 04:50:20.757664 (+ 195us) client_negotiation.cc:240] Negotiation successful
0809 04:50:20.757710 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":533,"threads_started":1}
I20260809 04:50:20.758316 7750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "535d374e28194260912c5e8167d4922e" candidate_uuid: "e4321eae55644c45a0f06238b1b6db9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eca8766fa25740338dd1e6fad45ebf4e" is_pre_election: true
I20260809 04:50:20.758749 7750 raft_consensus.cc:2454] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4321eae55644c45a0f06238b1b6db9f in term 0.
I20260809 04:50:20.759063 7546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "535d374e28194260912c5e8167d4922e" candidate_uuid: "e4321eae55644c45a0f06238b1b6db9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed195921b994cf589956abebb7bf697" is_pre_election: true
I20260809 04:50:20.759342 7858 leader_election.cc:302] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4321eae55644c45a0f06238b1b6db9f, eca8766fa25740338dd1e6fad45ebf4e; no voters:
I20260809 04:50:20.759555 7546 raft_consensus.cc:2454] T 535d374e28194260912c5e8167d4922e P fed195921b994cf589956abebb7bf697 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4321eae55644c45a0f06238b1b6db9f in term 0.
I20260809 04:50:20.759719 8103 raft_consensus.cc:2781] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:50:20.759855 8103 raft_consensus.cc:482] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:50:20.759971 8103 raft_consensus.cc:3037] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:20.762212 8103 raft_consensus.cc:504] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4321eae55644c45a0f06238b1b6db9f" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36297 } } peers { permanent_uuid: "fed195921b994cf589956abebb7bf697" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 40735 } } peers { permanent_uuid: "eca8766fa25740338dd1e6fad45ebf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43423 } }
I20260809 04:50:20.762868 8103 leader_election.cc:288] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [CANDIDATE]: Term 1 election: Requested vote from peers fed195921b994cf589956abebb7bf697 (127.31.93.1:40735), eca8766fa25740338dd1e6fad45ebf4e (127.31.93.2:43423)
I20260809 04:50:20.763182 7546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "535d374e28194260912c5e8167d4922e" candidate_uuid: "e4321eae55644c45a0f06238b1b6db9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed195921b994cf589956abebb7bf697"
I20260809 04:50:20.763362 7546 raft_consensus.cc:3037] T 535d374e28194260912c5e8167d4922e P fed195921b994cf589956abebb7bf697 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:20.763382 7750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "535d374e28194260912c5e8167d4922e" candidate_uuid: "e4321eae55644c45a0f06238b1b6db9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eca8766fa25740338dd1e6fad45ebf4e"
I20260809 04:50:20.763567 7750 raft_consensus.cc:3037] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:20.766579 7546 raft_consensus.cc:2454] T 535d374e28194260912c5e8167d4922e P fed195921b994cf589956abebb7bf697 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4321eae55644c45a0f06238b1b6db9f in term 1.
I20260809 04:50:20.766625 7750 raft_consensus.cc:2454] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4321eae55644c45a0f06238b1b6db9f in term 1.
I20260809 04:50:20.766968 7858 leader_election.cc:302] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4321eae55644c45a0f06238b1b6db9f, eca8766fa25740338dd1e6fad45ebf4e; no voters:
I20260809 04:50:20.767251 8103 raft_consensus.cc:2781] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:20.767691 8103 raft_consensus.cc:686] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [term 1 LEADER]: Becoming Leader. State: Replica: e4321eae55644c45a0f06238b1b6db9f, State: Running, Role: LEADER
I20260809 04:50:20.768195 8103 consensus_queue.cc:237] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4321eae55644c45a0f06238b1b6db9f" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36297 } } peers { permanent_uuid: "fed195921b994cf589956abebb7bf697" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 40735 } } peers { permanent_uuid: "eca8766fa25740338dd1e6fad45ebf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43423 } }
I20260809 04:50:20.772856 7324 catalog_manager.cc:5495] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f reported cstate change: term changed from 0 to 1, leader changed from <none> to e4321eae55644c45a0f06238b1b6db9f (127.31.93.3). New cstate: current_term: 1 leader_uuid: "e4321eae55644c45a0f06238b1b6db9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4321eae55644c45a0f06238b1b6db9f" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed195921b994cf589956abebb7bf697" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 40735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eca8766fa25740338dd1e6fad45ebf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43423 } health_report { overall_health: UNKNOWN } } }
I20260809 04:50:20.789738 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.787999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.31.93.3:36297)
0809 04:50:20.788140 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:20.788145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:20.788284 (+ 139us) server_negotiation.cc:407] Connection header received
0809 04:50:20.788429 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:20.788433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:20.788501 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:20.788580 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:20.789327 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:20.789331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:20.789334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:20.789349 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:20.789378 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:20.789390 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:20.789481 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:20.789528 (+ 47us) server_negotiation.cc:299] Negotiation successful
0809 04:50:20.789567 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 04:50:20.797401 7546 raft_consensus.cc:1261] T 535d374e28194260912c5e8167d4922e P fed195921b994cf589956abebb7bf697 [term 1 FOLLOWER]: Refusing update from remote peer e4321eae55644c45a0f06238b1b6db9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:20.797456 7750 raft_consensus.cc:1261] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e [term 1 FOLLOWER]: Refusing update from remote peer e4321eae55644c45a0f06238b1b6db9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:20.798377 8103 consensus_queue.cc:1035] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed195921b994cf589956abebb7bf697" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 40735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:20.798666 8109 consensus_queue.cc:1035] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eca8766fa25740338dd1e6fad45ebf4e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 04:50:20.799013 7612 tablet.cc:2369] T 535d374e28194260912c5e8167d4922e P fed195921b994cf589956abebb7bf697: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:50:20.818769 8116 mvcc.cc:205] Tried to move back new op lower bound from 7316484181176410112 to 7316484181070409728. Current Snapshot: MvccSnapshot[applied={T|T < 7316484181176410112}]
I20260809 04:50:20.825928 7325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:20.826139 7325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:20.829406 7325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4328ba20bbf040148cc46c4f0840e262: Sending DeleteTablet for 3 replicas of tablet 535d374e28194260912c5e8167d4922e
I20260809 04:50:20.830462 7938 tablet_service.cc:1514] Processing DeleteTablet for tablet 535d374e28194260912c5e8167d4922e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:20 UTC) from {username='slave'} at 127.0.0.1:41450
I20260809 04:50:20.830514 7519 tablet_service.cc:1514] Processing DeleteTablet for tablet 535d374e28194260912c5e8167d4922e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:20 UTC) from {username='slave'} at 127.0.0.1:49144
I20260809 04:50:20.831061 7715 tablet_service.cc:1514] Processing DeleteTablet for tablet 535d374e28194260912c5e8167d4922e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:20 UTC) from {username='slave'} at 127.0.0.1:42318
I20260809 04:50:20.832867 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 7419
I20260809 04:50:20.838730 8127 tablet_replica.cc:330] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f: stopping tablet replica
I20260809 04:50:20.839125 8127 raft_consensus.cc:2229] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:50:20.839715 8127 raft_consensus.cc:2258] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:20.841768 8128 tablet_replica.cc:330] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e: stopping tablet replica
I20260809 04:50:20.842584 8127 ts_tablet_manager.cc:1905] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:20.843185 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.834381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.31.93.1:40735)
0809 04:50:20.834553 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:20.834561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:20.840430 (+ 5869us) server_negotiation.cc:407] Connection header received
0809 04:50:20.840627 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:20.840633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:20.840742 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:20.840815 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:20.842340 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:20.842346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:20.842350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:20.842371 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:20.842408 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:20.842423 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:20.842511 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:20.842805 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 04:50:20.842867 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:50:20.844260 7602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:20.845458 8128 raft_consensus.cc:2229] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:20.845923 8128 raft_consensus.cc:2258] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:20.847963 8128 ts_tablet_manager.cc:1905] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:20.855978 8128 ts_tablet_manager.cc:1918] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:20.856141 8128 log.cc:1196] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/535d374e28194260912c5e8167d4922e
I20260809 04:50:20.856842 8128 ts_tablet_manager.cc:1939] T 535d374e28194260912c5e8167d4922e P eca8766fa25740338dd1e6fad45ebf4e: Deleting consensus metadata
I20260809 04:50:20.858604 7299 catalog_manager.cc:4841] TS eca8766fa25740338dd1e6fad45ebf4e (127.31.93.2:43423): tablet 535d374e28194260912c5e8167d4922e (table test-table [id=fdbbad1edb75490ea43d3322c3e3c517]) successfully deleted
I20260809 04:50:20.859872 8127 ts_tablet_manager.cc:1918] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:20.860072 8127 log.cc:1196] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/535d374e28194260912c5e8167d4922e
I20260809 04:50:20.860659 8127 ts_tablet_manager.cc:1939] T 535d374e28194260912c5e8167d4922e P e4321eae55644c45a0f06238b1b6db9f: Deleting consensus metadata
I20260809 04:50:20.862239 7298 catalog_manager.cc:4841] TS e4321eae55644c45a0f06238b1b6db9f (127.31.93.3:36297): tablet 535d374e28194260912c5e8167d4922e (table test-table [id=fdbbad1edb75490ea43d3322c3e3c517]) successfully deleted
I20260809 04:50:20.941767 8125 tablet_replica.cc:330] T 535d374e28194260912c5e8167d4922e P fed195921b994cf589956abebb7bf697: stopping tablet replica
I20260809 04:50:20.942008 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 7419
I20260809 04:50:20.964201 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 7615
W20260809 04:50:20.964071 7298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.93.1:40735 (error 108)
I20260809 04:50:20.966152 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.965619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:40735 (local address 127.0.0.1:49156)
0809 04:50:20.965839 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:20.966008 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260809 04:50:20.966475 7298 catalog_manager.cc:4584] TS fed195921b994cf589956abebb7bf697 (127.31.93.1:40735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 535d374e28194260912c5e8167d4922e: Network error: Client connection negotiation failed: client connection to 127.31.93.1:40735: connect: Connection refused (error 111)
I20260809 04:50:20.967768 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:20.965654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42332 (local address 127.31.93.2:43423)
0809 04:50:20.965856 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:20.965862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:20.965883 (+ 21us) server_negotiation.cc:407] Connection header received
0809 04:50:20.965947 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:20.965952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:20.966059 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:20.966172 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:20.967170 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:20.967177 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:20.967182 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:20.967202 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:20.967243 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:20.967259 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:20.967387 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:20.967435 (+ 48us) server_negotiation.cc:299] Negotiation successful
0809 04:50:20.967492 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 04:50:20.968516 7810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:21.020179 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:21.019623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:40735 (local address 127.0.0.1:49166)
0809 04:50:21.019888 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:21.019983 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260809 04:50:21.021341 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:21.020939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:40735 (local address 127.0.0.1:49168)
0809 04:50:21.021133 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:21.021210 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260809 04:50:21.058476 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 7615
I20260809 04:50:21.080227 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 7820
I20260809 04:50:21.081784 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:21.081236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:40735 (local address 127.0.0.1:49170)
0809 04:50:21.081468 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:21.081592 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260809 04:50:21.082950 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:21.082533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:40735 (local address 127.0.0.1:49180)
0809 04:50:21.082717 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:21.082806 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 04:50:21.083432 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:21.081088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.31.93.3:36297)
0809 04:50:21.081284 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:21.081290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:21.081537 (+ 247us) server_negotiation.cc:407] Connection header received
0809 04:50:21.081635 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:21.081638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:21.081697 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:21.081774 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:21.082798 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:21.082804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:21.082807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:21.082823 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:21.082852 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:21.082863 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:21.082981 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:21.083085 (+ 104us) server_negotiation.cc:299] Negotiation successful
0809 04:50:21.083144 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 04:50:21.084054 8053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:21.163790 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:21.163270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:40735 (local address 127.0.0.1:49190)
0809 04:50:21.163476 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:21.163597 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 04:50:21.164934 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:21.164461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:40735 (local address 127.0.0.1:49198)
0809 04:50:21.164704 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:21.164790 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260809 04:50:21.198674 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 7820
I20260809 04:50:21.222216 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 7270
I20260809 04:50:21.223354 7374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:21.325985 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 7270
2026-08-09T04:50:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2502 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260809 04:50:21.371598 32116 mini_oidc.cc:152] Starting JWKS server
I20260809 04:50:21.374480 32116 webserver.cc:460] Webserver started at https://127.0.0.1:43765/ using document root <none> and password file <none>
I20260809 04:50:21.374831 32116 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 04:50:21.375422 32116 webserver.cc:460] Webserver started at http://127.0.0.1:35003/ using document root <none> and password file <none>
2026-08-09T04:50:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:50:21Z Disabled control of system clock
I20260809 04:50:21.403057 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:32807
--webserver_interface=127.31.93.62
--webserver_port=0
--builtin_ntp_servers=127.31.93.20:41543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:32807
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43765/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/testchainca.cert with env {}
W20260809 04:50:21.712163 8153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:21.712461 8153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:21.712610 8153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:21.719532 8153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:21.719646 8153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:21.719712 8153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:21.719798 8153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:21.719861 8153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 04:50:21.719944 8153 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 04:50:21.720000 8153 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43765/jwks.json
I20260809 04:50:21.729543 8153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:41543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:32807
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:32807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43765/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:21.730806 8153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:21.732082 8153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:21.743937 8160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:21.744038 8164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:21.744417 8167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:21.744833 8153 server_base.cc:1034] running on GCE node
I20260809 04:50:21.745586 8153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:21.746974 8153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:21.748178 8153 hybrid_clock.cc:648] HybridClock initialized: now 1786251021748129 us; error 38 us; skew 500 ppm
I20260809 04:50:21.748497 8153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:50:21.750658 8153 webserver.cc:460] Webserver started at http://127.31.93.62:37123/ using document root <none> and password file <none>
I20260809 04:50:21.751305 8153 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:21.751416 8153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:21.751708 8153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:21.754513 8153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "45d367a27e1241c3a5ea0248eafa08ab"
format_stamp: "Formatted at 2026-08-09 04:50:21 on dist-test-slave-4tdj"
server_key: "c95b3442eb3e599c1c80e330aa7bbb97"
server_key_iv: "87ad36522fcdf0e8e3d9ec45d2581423"
server_key_version: "encryptionkey@0"
I20260809 04:50:21.755286 8153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "45d367a27e1241c3a5ea0248eafa08ab"
format_stamp: "Formatted at 2026-08-09 04:50:21 on dist-test-slave-4tdj"
server_key: "c95b3442eb3e599c1c80e330aa7bbb97"
server_key_iv: "87ad36522fcdf0e8e3d9ec45d2581423"
server_key_version: "encryptionkey@0"
I20260809 04:50:21.760176 8153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 04:50:21.763572 8173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:21.764853 8153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:50:21.765017 8153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "45d367a27e1241c3a5ea0248eafa08ab"
format_stamp: "Formatted at 2026-08-09 04:50:21 on dist-test-slave-4tdj"
server_key: "c95b3442eb3e599c1c80e330aa7bbb97"
server_key_iv: "87ad36522fcdf0e8e3d9ec45d2581423"
server_key_version: "encryptionkey@0"
I20260809 04:50:21.765213 8153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:21.789554 8153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:21.790346 8153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:21.790669 8153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:21.807313 8153 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:32807
I20260809 04:50:21.807284 8251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:32807 every 8 connection(s)
I20260809 04:50:21.808605 8153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:50:21.812196 8252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:21.817740 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 8153
I20260809 04:50:21.817919 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:50:21.818717 32116 external_mini_cluster.cc:1428] Setting key e3711e68c11473b636aac91a805191bd
I20260809 04:50:21.822384 8252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab: Bootstrap starting.
I20260809 04:50:21.825591 8252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:21.826790 8252 log.cc:824] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:21.829763 8252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab: No bootstrap required, opened a new log
I20260809 04:50:21.832006 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:21.820568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.31.93.62:32807)
0809 04:50:21.821500 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:21.821510 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:21.821550 (+ 40us) server_negotiation.cc:407] Connection header received
0809 04:50:21.823264 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:21.823285 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:21.823714 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:21.824017 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:21.824630 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:21.825938 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:21.827719 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:21.828456 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:21.828588 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:21.829780 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:21.829789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:21.829803 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:21.830385 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:21.830462 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:21.830483 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:21.830567 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:21.830778 (+ 211us) server_negotiation.cc:299] Negotiation successful
0809 04:50:21.831055 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":332,"threads_started":1}
I20260809 04:50:21.835532 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:50:21.837333 8252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d367a27e1241c3a5ea0248eafa08ab" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 32807 } }
I20260809 04:50:21.837725 8252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:21.837819 8252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d367a27e1241c3a5ea0248eafa08ab, State: Initialized, Role: FOLLOWER
I20260809 04:50:21.838472 8252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d367a27e1241c3a5ea0248eafa08ab" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 32807 } }
I20260809 04:50:21.838739 8252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:50:21.838886 8252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:50:21.839032 8252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:21.841002 8252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d367a27e1241c3a5ea0248eafa08ab" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 32807 } }
I20260809 04:50:21.841432 8252 leader_election.cc:302] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45d367a27e1241c3a5ea0248eafa08ab; no voters:
I20260809 04:50:21.841920 8252 leader_election.cc:288] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:50:21.842022 8259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:21.842782 8259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [term 1 LEADER]: Becoming Leader. State: Replica: 45d367a27e1241c3a5ea0248eafa08ab, State: Running, Role: LEADER
I20260809 04:50:21.843307 8259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d367a27e1241c3a5ea0248eafa08ab" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 32807 } }
I20260809 04:50:21.844050 8252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:21.846148 8264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45d367a27e1241c3a5ea0248eafa08ab. Latest consensus state: current_term: 1 leader_uuid: "45d367a27e1241c3a5ea0248eafa08ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d367a27e1241c3a5ea0248eafa08ab" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 32807 } } }
I20260809 04:50:21.846617 8264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:21.847100 8260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45d367a27e1241c3a5ea0248eafa08ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d367a27e1241c3a5ea0248eafa08ab" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 32807 } } }
I20260809 04:50:21.847333 8260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:21.849983 8271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:50:21.853895 8271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:50:21.861515 8271 catalog_manager.cc:1347] Generated new cluster ID: ff6493d6f3a74178b5fb69db1263fe0c
I20260809 04:50:21.861610 8271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:50:21.878173 8271 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:50:21.879357 8271 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:50:21.886193 8271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45d367a27e1241c3a5ea0248eafa08ab: Generated new TSK 0
I20260809 04:50:21.887120 8271 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 04:50:21.933696 8304 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260809 04:50:21.933884 8304 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 04:50:21.936165 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:21.912299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.31.93.62:32807)
0809 04:50:21.912479 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:21.912485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:21.912947 (+ 462us) server_negotiation.cc:407] Connection header received
0809 04:50:21.913236 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:21.913241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:21.913324 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:21.913475 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0809 04:50:21.913956 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:21.915116 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:21.916501 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:21.917176 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:21.917302 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:21.917847 (+ 545us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 04:50:21.935524 (+ 17677us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43765/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 04:50:21.935872 (+ 348us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42566: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43765/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260809 04:50:21.936287 8256 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42566: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43765/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260809 04:50:21.937858 8303 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:21.912189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:32807 (local address 127.0.0.1:42566)
0809 04:50:21.912857 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:21.912873 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:21.913005 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:21.913488 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:21.913492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:21.913501 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0809 04:50:21.913824 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:21.913832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:21.915233 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:21.915237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:21.916366 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:21.916378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:21.917323 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:21.917327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:21.917442 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:21.917493 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 04:50:21.935778 (+ 18285us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43765/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 04:50:21.936014 (+ 236us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.93.62:32807: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43765/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":347,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.31.93.62:32807: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43765/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260809 04:50:21.939425 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 8153
I20260809 04:50:21.940420 8245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:22.030988 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 8153
2026-08-09T04:50:22Z chronyd exiting
I20260809 04:50:22.084824 32116 test_util.cc:175] -----------------------------------------------
I20260809 04:50:22.084966 32116 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786250977734564-32116-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (719 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260809 04:50:22.088613 32116 mini_oidc.cc:152] Starting JWKS server
I20260809 04:50:22.090399 32116 webserver.cc:460] Webserver started at https://127.0.0.1:40465/ using document root <none> and password file <none>
I20260809 04:50:22.090628 32116 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 04:50:22.091342 32116 webserver.cc:460] Webserver started at http://127.0.0.1:44575/ using document root <none> and password file <none>
2026-08-09T04:50:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:50:22Z Disabled control of system clock
I20260809 04:50:22.119088 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:39395
--webserver_interface=127.31.93.62
--webserver_port=0
--builtin_ntp_servers=127.31.93.20:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:39395
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40465/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/testchainca.cert with env {}
W20260809 04:50:22.428503 8317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:22.428771 8317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:22.428895 8317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:22.435822 8317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:22.435918 8317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:22.435982 8317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:22.436055 8317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:22.436115 8317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 04:50:22.436198 8317 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 04:50:22.436244 8317 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40465/jwks.json
I20260809 04:50:22.445626 8317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:39395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:39395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40465/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:22.446775 8317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:22.447990 8317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:22.458576 8323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:22.458527 8324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:22.458608 8328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:22.459622 8317 server_base.cc:1034] running on GCE node
I20260809 04:50:22.460281 8317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:22.461467 8317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:22.462659 8317 hybrid_clock.cc:648] HybridClock initialized: now 1786251022462615 us; error 37 us; skew 500 ppm
I20260809 04:50:22.462965 8317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:50:22.464944 8317 webserver.cc:460] Webserver started at http://127.31.93.62:46459/ using document root <none> and password file <none>
I20260809 04:50:22.465567 8317 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:22.465667 8317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:22.465965 8317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:22.468616 8317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "08393d509ed94b698a717df371a24bf1"
format_stamp: "Formatted at 2026-08-09 04:50:22 on dist-test-slave-4tdj"
server_key: "b29736965da6f53b38b8daeea6042d9e"
server_key_iv: "fd57728bb31642e809fc52eb50037ac4"
server_key_version: "encryptionkey@0"
I20260809 04:50:22.469332 8317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "08393d509ed94b698a717df371a24bf1"
format_stamp: "Formatted at 2026-08-09 04:50:22 on dist-test-slave-4tdj"
server_key: "b29736965da6f53b38b8daeea6042d9e"
server_key_iv: "fd57728bb31642e809fc52eb50037ac4"
server_key_version: "encryptionkey@0"
I20260809 04:50:22.473973 8317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 04:50:22.477191 8335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:22.478634 8317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:50:22.478817 8317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "08393d509ed94b698a717df371a24bf1"
format_stamp: "Formatted at 2026-08-09 04:50:22 on dist-test-slave-4tdj"
server_key: "b29736965da6f53b38b8daeea6042d9e"
server_key_iv: "fd57728bb31642e809fc52eb50037ac4"
server_key_version: "encryptionkey@0"
I20260809 04:50:22.479007 8317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:22.496061 8317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:22.496773 8317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:22.497036 8317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:22.513409 8317 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:39395
I20260809 04:50:22.513432 8412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:39395 every 8 connection(s)
I20260809 04:50:22.514611 8317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:50:22.515743 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 8317
I20260809 04:50:22.515947 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:50:22.516803 32116 external_mini_cluster.cc:1428] Setting key 98bd1cbc778cdf111292f0c48c2e07b4
I20260809 04:50:22.518965 8414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:22.529466 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:22.518639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.31.93.62:39395)
0809 04:50:22.519564 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:22.519574 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:22.519613 (+ 39us) server_negotiation.cc:407] Connection header received
0809 04:50:22.521098 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:22.521117 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:22.521546 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:22.521857 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:22.522411 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:22.523673 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:22.525417 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:22.526017 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:22.526144 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:22.527338 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:22.527345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:22.527360 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:22.527860 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:22.527934 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:22.527964 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:22.528050 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:22.528298 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 04:50:22.528588 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":351,"threads_started":1}
I20260809 04:50:22.530915 8414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1: Bootstrap starting.
I20260809 04:50:22.532996 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:50:22.534121 8414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:22.535113 8414 log.cc:824] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:22.537742 8414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1: No bootstrap required, opened a new log
I20260809 04:50:22.544752 8414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08393d509ed94b698a717df371a24bf1" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 39395 } }
I20260809 04:50:22.545040 8414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:22.545117 8414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08393d509ed94b698a717df371a24bf1, State: Initialized, Role: FOLLOWER
I20260809 04:50:22.545629 8414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08393d509ed94b698a717df371a24bf1" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 39395 } }
I20260809 04:50:22.545825 8414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:50:22.545918 8414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:50:22.546046 8414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:22.548192 8414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08393d509ed94b698a717df371a24bf1" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 39395 } }
I20260809 04:50:22.548622 8414 leader_election.cc:302] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08393d509ed94b698a717df371a24bf1; no voters:
I20260809 04:50:22.549079 8414 leader_election.cc:288] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:50:22.549212 8419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:22.549883 8419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [term 1 LEADER]: Becoming Leader. State: Replica: 08393d509ed94b698a717df371a24bf1, State: Running, Role: LEADER
I20260809 04:50:22.550401 8419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08393d509ed94b698a717df371a24bf1" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 39395 } }
I20260809 04:50:22.550957 8414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:22.552937 8422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08393d509ed94b698a717df371a24bf1. Latest consensus state: current_term: 1 leader_uuid: "08393d509ed94b698a717df371a24bf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08393d509ed94b698a717df371a24bf1" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 39395 } } }
I20260809 04:50:22.553326 8422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:22.553157 8420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08393d509ed94b698a717df371a24bf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08393d509ed94b698a717df371a24bf1" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 39395 } } }
I20260809 04:50:22.553467 8420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1 [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:22.555063 8435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:50:22.559338 8435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:50:22.567445 8435 catalog_manager.cc:1347] Generated new cluster ID: 918e0ef486b442ecb5bccfdc3e1f767f
I20260809 04:50:22.567571 8435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:50:22.587906 8435 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:50:22.589068 8435 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:50:22.597841 8435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08393d509ed94b698a717df371a24bf1: Generated new TSK 0
I20260809 04:50:22.598590 8435 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 04:50:22.636664 8463 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260809 04:50:22.636775 8463 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 04:50:22.637658 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:22.618940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59554 (local address 127.31.93.62:39395)
0809 04:50:22.619152 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:22.619158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:22.619520 (+ 362us) server_negotiation.cc:407] Connection header received
0809 04:50:22.619743 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:22.619747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:22.619808 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:22.619934 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0809 04:50:22.620509 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:22.621221 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:22.623163 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:22.623587 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:22.623673 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:22.624185 (+ 512us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 04:50:22.637178 (+ 12993us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 04:50:22.637436 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59554: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 04:50:22.637777 8417 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59554: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260809 04:50:22.638278 8460 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:22.618844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:39395 (local address 127.0.0.1:59554)
0809 04:50:22.619423 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:22.619442 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:22.619595 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:22.619964 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:22.619969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:22.619982 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0809 04:50:22.620326 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:22.620338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:22.621351 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:22.621357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:22.623030 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:22.623046 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:22.623731 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:22.623737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:22.623929 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:22.623971 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 04:50:22.637369 (+ 13398us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 04:50:22.637617 (+ 248us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.93.62:39395: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":290,"threads_started":1}
I20260809 04:50:22.639314 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 8317
I20260809 04:50:22.640312 8406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:22.730077 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 8317
2026-08-09T04:50:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (702 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260809 04:50:22.791528 32116 mini_oidc.cc:152] Starting JWKS server
I20260809 04:50:22.793347 32116 webserver.cc:460] Webserver started at https://127.0.0.1:41749/ using document root <none> and password file <none>
I20260809 04:50:22.793560 32116 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 04:50:22.794106 32116 webserver.cc:460] Webserver started at http://127.0.0.1:43873/ using document root <none> and password file <none>
2026-08-09T04:50:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:50:22Z Disabled control of system clock
I20260809 04:50:22.821568 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:36039
--webserver_interface=127.31.93.62
--webserver_port=0
--builtin_ntp_servers=127.31.93.20:41733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:36039
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41749/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 04:50:23.143767 8474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:23.144052 8474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:23.144155 8474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:23.151254 8474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:23.151360 8474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:23.151412 8474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:23.151480 8474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:23.151530 8474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 04:50:23.151600 8474 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 04:50:23.151650 8474 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41749/jwks.json
I20260809 04:50:23.160573 8474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:41733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:36039
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:36039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41749/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:23.161760 8474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:23.163544 8474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:23.178583 8484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:23.178627 8481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:23.178543 8482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:23.180014 8474 server_base.cc:1034] running on GCE node
I20260809 04:50:23.180732 8474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:23.181993 8474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:23.183195 8474 hybrid_clock.cc:648] HybridClock initialized: now 1786251023183142 us; error 40 us; skew 500 ppm
I20260809 04:50:23.183524 8474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:50:23.189901 8474 webserver.cc:460] Webserver started at http://127.31.93.62:34591/ using document root <none> and password file <none>
I20260809 04:50:23.190622 8474 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:23.190725 8474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:23.191030 8474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:23.193789 8474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "e33fe803d2f946a7bf7229bab5f3d367"
format_stamp: "Formatted at 2026-08-09 04:50:23 on dist-test-slave-4tdj"
server_key: "c2bd5db5885627bbac8ff90f42fa392b"
server_key_iv: "4e9eb8a934739a30e1307e01929db624"
server_key_version: "encryptionkey@0"
I20260809 04:50:23.194597 8474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "e33fe803d2f946a7bf7229bab5f3d367"
format_stamp: "Formatted at 2026-08-09 04:50:23 on dist-test-slave-4tdj"
server_key: "c2bd5db5885627bbac8ff90f42fa392b"
server_key_iv: "4e9eb8a934739a30e1307e01929db624"
server_key_version: "encryptionkey@0"
I20260809 04:50:23.199366 8474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:50:23.202656 8493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:23.203883 8474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 04:50:23.204061 8474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "e33fe803d2f946a7bf7229bab5f3d367"
format_stamp: "Formatted at 2026-08-09 04:50:23 on dist-test-slave-4tdj"
server_key: "c2bd5db5885627bbac8ff90f42fa392b"
server_key_iv: "4e9eb8a934739a30e1307e01929db624"
server_key_version: "encryptionkey@0"
I20260809 04:50:23.204277 8474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:23.217062 8474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:23.217820 8474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:23.218125 8474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:23.236284 8474 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:36039
I20260809 04:50:23.236304 8567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:36039 every 8 connection(s)
I20260809 04:50:23.237605 8474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:50:23.241120 8568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:23.246955 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 8474
I20260809 04:50:23.247180 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:50:23.247953 32116 external_mini_cluster.cc:1428] Setting key e897779fa27c0d9186a5d32568d01301
I20260809 04:50:23.253023 8568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367: Bootstrap starting.
I20260809 04:50:23.256428 8568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:23.257568 8568 log.cc:824] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:23.260651 8568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367: No bootstrap required, opened a new log
I20260809 04:50:23.264533 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:23.250003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.31.93.62:36039)
0809 04:50:23.251238 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:23.251257 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:23.253735 (+ 2478us) server_negotiation.cc:407] Connection header received
0809 04:50:23.255575 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:23.255595 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:23.256012 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:23.256318 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:23.256902 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:23.258219 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:23.259648 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:23.260634 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:23.260820 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:23.261853 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:23.261864 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:23.261881 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:23.262592 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:23.262691 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:23.262720 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:23.262850 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:23.262994 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 04:50:23.263306 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":433,"threads_started":1}
I20260809 04:50:23.268353 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:50:23.270347 8568 raft_consensus.cc:348] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e33fe803d2f946a7bf7229bab5f3d367" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36039 } }
I20260809 04:50:23.270758 8568 raft_consensus.cc:374] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:23.270864 8568 raft_consensus.cc:729] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e33fe803d2f946a7bf7229bab5f3d367, State: Initialized, Role: FOLLOWER
I20260809 04:50:23.271471 8568 consensus_queue.cc:260] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e33fe803d2f946a7bf7229bab5f3d367" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36039 } }
I20260809 04:50:23.271780 8568 raft_consensus.cc:388] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:50:23.271921 8568 raft_consensus.cc:482] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:50:23.272110 8568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:23.274215 8568 raft_consensus.cc:504] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e33fe803d2f946a7bf7229bab5f3d367" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36039 } }
I20260809 04:50:23.274719 8568 leader_election.cc:302] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e33fe803d2f946a7bf7229bab5f3d367; no voters:
I20260809 04:50:23.275267 8568 leader_election.cc:288] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:50:23.275399 8575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:23.276094 8575 raft_consensus.cc:686] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [term 1 LEADER]: Becoming Leader. State: Replica: e33fe803d2f946a7bf7229bab5f3d367, State: Running, Role: LEADER
I20260809 04:50:23.276602 8575 consensus_queue.cc:237] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e33fe803d2f946a7bf7229bab5f3d367" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36039 } }
I20260809 04:50:23.277144 8568 sys_catalog.cc:564] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:23.279066 8578 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e33fe803d2f946a7bf7229bab5f3d367. Latest consensus state: current_term: 1 leader_uuid: "e33fe803d2f946a7bf7229bab5f3d367" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e33fe803d2f946a7bf7229bab5f3d367" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36039 } } }
I20260809 04:50:23.279436 8578 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:23.280316 8582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:50:23.280174 8576 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e33fe803d2f946a7bf7229bab5f3d367" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e33fe803d2f946a7bf7229bab5f3d367" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36039 } } }
I20260809 04:50:23.285146 8576 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367 [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:23.287951 8582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:50:23.296824 8582 catalog_manager.cc:1347] Generated new cluster ID: 6d3d14eacb354900a3e0532a89c59cf8
I20260809 04:50:23.296932 8582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:50:23.308784 8582 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:50:23.309894 8582 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:50:23.323359 8582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e33fe803d2f946a7bf7229bab5f3d367: Generated new TSK 0
I20260809 04:50:23.324227 8582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:50:23.353597 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:23.345597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45310 (local address 127.31.93.62:36039)
0809 04:50:23.345779 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:23.345784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:23.346106 (+ 322us) server_negotiation.cc:407] Connection header received
0809 04:50:23.346338 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:23.346344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:23.346435 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:23.346602 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0809 04:50:23.347000 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:23.348111 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:23.349381 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:23.350057 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:23.350175 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:23.350658 (+ 483us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 04:50:23.352967 (+ 2309us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41749/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 04:50:23.353293 (+ 326us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45310: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41749/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 04:50:23.353724 8572 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45310: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41749/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260809 04:50:23.355293 8615 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:23.345464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36039 (local address 127.0.0.1:45310)
0809 04:50:23.346015 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:23.346031 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:23.346141 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:23.346601 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:23.346605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:23.346614 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0809 04:50:23.346863 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:23.346871 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:23.348222 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:23.348225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:23.349256 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:23.349267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:23.350220 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:23.350224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:23.350391 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:23.350424 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 04:50:23.353174 (+ 2750us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41749/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 04:50:23.354889 (+ 1715us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.93.62:36039: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41749/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":1444,"thread_start_us":292,"threads_started":1}
I20260809 04:50:23.356364 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 8474
I20260809 04:50:23.357443 8562 generic_service.cc:196] Checking for leaks (request via RPC)
E20260809 04:50:23.364509 8617 webserver.cc:533] Webserver: sslize failed:
E20260809 04:50:23.364596 8617 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 04:50:23.447474 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 8474
2026-08-09T04:50:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (717 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-09T04:50:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:50:23Z Disabled control of system clock
I20260809 04:50:23.539484 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:41481
--webserver_interface=127.31.93.62
--webserver_port=0
--builtin_ntp_servers=127.31.93.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:41481
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/testchainca.cert with env {}
W20260809 04:50:23.847102 8625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:23.847394 8625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:23.847530 8625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:23.854599 8625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:23.854700 8625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:23.854763 8625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:23.854848 8625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:23.854907 8625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 04:50:23.854990 8625 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 04:50:23.855042 8625 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260809 04:50:23.864619 8625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:41481
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:41481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:23.865862 8625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:23.867142 8625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:23.879004 8632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:23.878955 8634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:23.878906 8637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:23.880053 8625 server_base.cc:1034] running on GCE node
I20260809 04:50:23.880753 8625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:23.882016 8625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:23.883219 8625 hybrid_clock.cc:648] HybridClock initialized: now 1786251023883174 us; error 34 us; skew 500 ppm
I20260809 04:50:23.883567 8625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:50:23.885643 8625 webserver.cc:460] Webserver started at http://127.31.93.62:37315/ using document root <none> and password file <none>
I20260809 04:50:23.886267 8625 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:23.886435 8625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:23.886726 8625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:23.889374 8625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "7a8954db2efe48e495388a090aad9cf0"
format_stamp: "Formatted at 2026-08-09 04:50:23 on dist-test-slave-4tdj"
server_key: "1db605daad32cc3fbf6d19c5845caada"
server_key_iv: "85ce3bb3dde0157723c2a7ccff7037a3"
server_key_version: "encryptionkey@0"
I20260809 04:50:23.890105 8625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "7a8954db2efe48e495388a090aad9cf0"
format_stamp: "Formatted at 2026-08-09 04:50:23 on dist-test-slave-4tdj"
server_key: "1db605daad32cc3fbf6d19c5845caada"
server_key_iv: "85ce3bb3dde0157723c2a7ccff7037a3"
server_key_version: "encryptionkey@0"
I20260809 04:50:23.894843 8625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260809 04:50:23.897902 8643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:23.899113 8625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:50:23.899269 8625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "7a8954db2efe48e495388a090aad9cf0"
format_stamp: "Formatted at 2026-08-09 04:50:23 on dist-test-slave-4tdj"
server_key: "1db605daad32cc3fbf6d19c5845caada"
server_key_iv: "85ce3bb3dde0157723c2a7ccff7037a3"
server_key_version: "encryptionkey@0"
I20260809 04:50:23.899458 8625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:23.920619 8625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:23.921346 8625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:23.921607 8625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:23.938038 8625 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:41481
I20260809 04:50:23.938058 8729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:41481 every 8 connection(s)
I20260809 04:50:23.939248 8625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:50:23.942682 8731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:23.943652 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 8625
I20260809 04:50:23.943820 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:50:23.944478 32116 external_mini_cluster.cc:1428] Setting key 379c2ff08718e615954733efae7680f0
I20260809 04:50:23.956349 8731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0: Bootstrap starting.
I20260809 04:50:23.958240 8738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:23.946273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49704 (local address 127.31.93.62:41481)
0809 04:50:23.947358 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:23.947375 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:23.947425 (+ 50us) server_negotiation.cc:407] Connection header received
0809 04:50:23.948957 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:23.948988 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:23.949533 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:23.949955 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:23.950463 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:23.952248 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:23.953440 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:23.954490 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:23.954662 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:23.955673 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:23.955683 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:23.955699 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:50:23.956348 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:23.956441 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:50:23.956466 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:23.956570 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:23.956757 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 04:50:23.957040 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":323,"threads_started":1}
I20260809 04:50:23.959784 8731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:23.960855 8731 log.cc:824] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:23.961702 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:50:23.963804 8731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0: No bootstrap required, opened a new log
I20260809 04:50:23.971051 8731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8954db2efe48e495388a090aad9cf0" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41481 } }
I20260809 04:50:23.971364 8731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:23.971437 8731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8954db2efe48e495388a090aad9cf0, State: Initialized, Role: FOLLOWER
I20260809 04:50:23.971904 8731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8954db2efe48e495388a090aad9cf0" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41481 } }
I20260809 04:50:23.972096 8731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:50:23.972194 8731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:50:23.972311 8731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:23.974200 8731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8954db2efe48e495388a090aad9cf0" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41481 } }
I20260809 04:50:23.974754 8731 leader_election.cc:302] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a8954db2efe48e495388a090aad9cf0; no voters:
I20260809 04:50:23.975276 8731 leader_election.cc:288] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:50:23.975427 8741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:23.976117 8741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8954db2efe48e495388a090aad9cf0, State: Running, Role: LEADER
I20260809 04:50:23.976635 8741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8954db2efe48e495388a090aad9cf0" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41481 } }
I20260809 04:50:23.977103 8731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:23.979408 8743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a8954db2efe48e495388a090aad9cf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8954db2efe48e495388a090aad9cf0" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41481 } } }
I20260809 04:50:23.979773 8743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:23.980476 8744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a8954db2efe48e495388a090aad9cf0. Latest consensus state: current_term: 1 leader_uuid: "7a8954db2efe48e495388a090aad9cf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8954db2efe48e495388a090aad9cf0" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41481 } } }
I20260809 04:50:23.980795 8744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0 [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:23.983376 8753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:50:23.988266 8753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:50:23.995654 8753 catalog_manager.cc:1347] Generated new cluster ID: 591ab4e0078a438586653175a868c32d
I20260809 04:50:23.995754 8753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:50:24.005493 8753 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:50:24.007179 8753 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:50:24.019901 8753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a8954db2efe48e495388a090aad9cf0: Generated new TSK 0
I20260809 04:50:24.020596 8753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:50:24.075253 8738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:24.030838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49720 (local address 127.31.93.62:41481)
0809 04:50:24.031071 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:24.031078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:24.031557 (+ 479us) server_negotiation.cc:407] Connection header received
0809 04:50:24.031786 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:24.031790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:24.031859 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:24.031980 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0809 04:50:24.032519 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:24.033216 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:24.035219 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:24.035650 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:24.035741 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:24.036305 (+ 564us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 04:50:24.074698 (+ 38393us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 04:50:24.075014 (+ 316us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49720: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":74}
W20260809 04:50:24.075383 8738 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49720: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260809 04:50:24.075789 8771 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:24.030723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41481 (local address 127.0.0.1:49720)
0809 04:50:24.031438 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:24.031460 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:24.031627 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:24.032022 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:24.032028 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:24.032042 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0809 04:50:24.032364 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:24.032377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:24.033360 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:24.033366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:24.035077 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:24.035094 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:24.035822 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:24.035829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:24.036043 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:24.036092 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 04:50:24.074980 (+ 38888us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 04:50:24.075311 (+ 331us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.93.62:41481: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":405,"threads_started":1}
I20260809 04:50:24.077370 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 8625
I20260809 04:50:24.078588 8723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:24.170328 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 8625
2026-08-09T04:50:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (701 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (400 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (395 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:50:25 dist-test-slave-4tdj krb5kdc[8793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:50:25 dist-test-slave-4tdj krb5kdc[8793](info): set up 2 sockets
Aug 09 04:50:25 dist-test-slave-4tdj krb5kdc[8793](info): commencing operation
krb5kdc: starting...
W20260809 04:50:26.163580 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:50:26 dist-test-slave-4tdj krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:50:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:50:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:50:26.355139 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:38019
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:38019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:26.663527 8816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:26.663790 8816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:26.663916 8816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:26.670821 8816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:26.670921 8816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:26.671028 8816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:26.671108 8816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:26.671161 8816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:50:26.680544 8816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:38019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:38019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:26.681715 8816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:26.682937 8816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:26.693944 8825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:26.693884 8830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:26.693913 8826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:26.694738 8816 server_base.cc:1034] running on GCE node
I20260809 04:50:26.695422 8816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:26.696672 8816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:26.697875 8816 hybrid_clock.cc:648] HybridClock initialized: now 1786251026697818 us; error 43 us; skew 500 ppm
I20260809 04:50:26.698186 8816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:26 dist-test-slave-4tdj krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251026, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:26.701186 8816 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:50:26.702515 8816 webserver.cc:460] Webserver started at http://127.31.93.62:43635/ using document root <none> and password file <none>
I20260809 04:50:26.703128 8816 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:26.703239 8816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:26.703529 8816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:26.706334 8816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "36a96c72de714e47a854be394d26c2ad"
format_stamp: "Formatted at 2026-08-09 04:50:26 on dist-test-slave-4tdj"
server_key: "cc0a93fdb702dfe4bfffc4f71ea13580"
server_key_iv: "4d8ebf3dfb42cceb741ea21918dab950"
server_key_version: "encryptionkey@0"
I20260809 04:50:26.707100 8816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "36a96c72de714e47a854be394d26c2ad"
format_stamp: "Formatted at 2026-08-09 04:50:26 on dist-test-slave-4tdj"
server_key: "cc0a93fdb702dfe4bfffc4f71ea13580"
server_key_iv: "4d8ebf3dfb42cceb741ea21918dab950"
server_key_version: "encryptionkey@0"
I20260809 04:50:26.711885 8816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260809 04:50:26.715011 8839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:26.716194 8816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:50:26.716346 8816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "36a96c72de714e47a854be394d26c2ad"
format_stamp: "Formatted at 2026-08-09 04:50:26 on dist-test-slave-4tdj"
server_key: "cc0a93fdb702dfe4bfffc4f71ea13580"
server_key_iv: "4d8ebf3dfb42cceb741ea21918dab950"
server_key_version: "encryptionkey@0"
I20260809 04:50:26.716504 8816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:26.741842 8816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:26.745781 8816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:26.746032 8816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:26.762181 8816 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:38019
I20260809 04:50:26.762215 8922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:38019 every 8 connection(s)
I20260809 04:50:26.763406 8816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:50:26.766878 8923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:26.771674 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 8816
I20260809 04:50:26.771845 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:50:26.772486 32116 external_mini_cluster.cc:1428] Setting key e620b9d79d28f5ce95d5eedd348b1faa
I20260809 04:50:26.777906 8923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad: Bootstrap starting.
I20260809 04:50:26.781433 8923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:26.782492 8923 log.cc:824] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:26.785143 8923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad: No bootstrap required, opened a new log
Aug 09 04:50:26 dist-test-slave-4tdj krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:26.794631 8923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a96c72de714e47a854be394d26c2ad" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 38019 } }
I20260809 04:50:26.795090 8923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:26.795198 8923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a96c72de714e47a854be394d26c2ad, State: Initialized, Role: FOLLOWER
I20260809 04:50:26.795656 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:26.774381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40616 (local address 127.31.93.62:38019)
0809 04:50:26.775211 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:26.775225 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:26.775275 (+ 50us) server_negotiation.cc:407] Connection header received
0809 04:50:26.777552 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:26.777589 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:26.778040 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:26.778463 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:26.779715 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:26.781612 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:26.782903 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:26.783671 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:26.783859 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:26.787545 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:26.787557 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:26.787575 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:26.787626 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:26.791419 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:26.792055 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:26.792063 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:26.792072 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:26.792297 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:26.792642 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:26.792647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:26.792650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:26.793328 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:26.793649 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:26.793919 (+ 270us) server_negotiation.cc:299] Negotiation successful
0809 04:50:26.794352 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":324,"threads_started":1}
I20260809 04:50:26.795845 8923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a96c72de714e47a854be394d26c2ad" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 38019 } }
I20260809 04:50:26.796132 8923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:50:26.796272 8923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:50:26.796432 8923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:26.799381 8923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a96c72de714e47a854be394d26c2ad" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 38019 } }
I20260809 04:50:26.799839 8923 leader_election.cc:302] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36a96c72de714e47a854be394d26c2ad; no voters:
I20260809 04:50:26.800362 8923 leader_election.cc:288] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:50:26.800462 8933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:26.801517 8933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [term 1 LEADER]: Becoming Leader. State: Replica: 36a96c72de714e47a854be394d26c2ad, State: Running, Role: LEADER
I20260809 04:50:26.802083 8933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a96c72de714e47a854be394d26c2ad" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 38019 } }
I20260809 04:50:26.802376 8923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:26.804864 8934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36a96c72de714e47a854be394d26c2ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a96c72de714e47a854be394d26c2ad" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 38019 } } }
I20260809 04:50:26.805230 8934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:26.805199 8935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36a96c72de714e47a854be394d26c2ad. Latest consensus state: current_term: 1 leader_uuid: "36a96c72de714e47a854be394d26c2ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a96c72de714e47a854be394d26c2ad" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 38019 } } }
I20260809 04:50:26.806113 8935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:26.809988 8946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:50:26.813922 8946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:50:26.821404 8946 catalog_manager.cc:1347] Generated new cluster ID: dc15cae0256d471585087a9c67bd7c0f
I20260809 04:50:26.821496 8946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:50:26.850103 8946 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:50:26.851374 8946 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:50:26.858254 8946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36a96c72de714e47a854be394d26c2ad: Generated new TSK 0
I20260809 04:50:26.859021 8946 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:50:26.915611 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:38019
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:27.248183 8968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:27.248420 8968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:27.248514 8968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:27.255338 8968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:27.255434 8968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:27.255544 8968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:50:27.264580 8968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:38019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:27.265694 8968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:27.266930 8968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:27.278949 8978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:27.279052 8981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:27.279079 8979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:27.280076 8968 server_base.cc:1034] running on GCE node
I20260809 04:50:27.280635 8968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:27.281469 8968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:27.282670 8968 hybrid_clock.cc:648] HybridClock initialized: now 1786251027282623 us; error 41 us; skew 500 ppm
I20260809 04:50:27.282949 8968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:27 dist-test-slave-4tdj krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251027, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:27.285840 8968 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:50:27.287134 8968 webserver.cc:460] Webserver started at http://127.31.93.1:45573/ using document root <none> and password file <none>
I20260809 04:50:27.287766 8968 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:27.287880 8968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:27.288162 8968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:27.290911 8968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "a194bff1512e4e3f98186db10c080b28"
format_stamp: "Formatted at 2026-08-09 04:50:27 on dist-test-slave-4tdj"
server_key: "3c9a044712df4bce460fab86c24603b4"
server_key_iv: "0ae822c3734e73cd3e235fb1aaf669e4"
server_key_version: "encryptionkey@0"
I20260809 04:50:27.291628 8968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "a194bff1512e4e3f98186db10c080b28"
format_stamp: "Formatted at 2026-08-09 04:50:27 on dist-test-slave-4tdj"
server_key: "3c9a044712df4bce460fab86c24603b4"
server_key_iv: "0ae822c3734e73cd3e235fb1aaf669e4"
server_key_version: "encryptionkey@0"
I20260809 04:50:27.296375 8968 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:50:27.299530 8988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:27.300719 8968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:50:27.300863 8968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "a194bff1512e4e3f98186db10c080b28"
format_stamp: "Formatted at 2026-08-09 04:50:27 on dist-test-slave-4tdj"
server_key: "3c9a044712df4bce460fab86c24603b4"
server_key_iv: "0ae822c3734e73cd3e235fb1aaf669e4"
server_key_version: "encryptionkey@0"
I20260809 04:50:27.301028 8968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:27.339323 8968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:27.342754 8968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:27.343003 8968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:27.343848 8968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:27.345058 8968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:27.345134 8968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:27.345261 8968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:27.345316 8968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:27.373335 8968 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:44813
I20260809 04:50:27.373400 9174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:44813 every 8 connection(s)
I20260809 04:50:27.374523 8968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:50:27.376582 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 8968
I20260809 04:50:27.376761 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:50:27.377379 32116 external_mini_cluster.cc:1428] Setting key 16b02e6d38f561e46c2581ace86c299e
Aug 09 04:50:27 dist-test-slave-4tdj krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251027, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:27.398798 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:27.376786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:43979 (local address 127.31.93.62:38019)
0809 04:50:27.383565 (+ 6779us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:27.383569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:27.383586 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:50:27.385180 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:27.385183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:27.385258 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:27.385344 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:27.387080 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:27.388049 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:27.389472 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:27.389873 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:27.389972 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:27.393572 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:27.393577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:27.393580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:27.393610 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:27.395971 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:27.396630 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:27.396634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:27.396636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:27.396758 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:27.397145 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:27.397148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:27.397151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:27.397404 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:27.397531 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:27.398185 (+ 654us) server_negotiation.cc:299] Negotiation successful
0809 04:50:27.398448 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6672,"thread_start_us":6580,"threads_started":1}
I20260809 04:50:27.399664 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:27.377354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:38019 (local address 127.31.93.1:43979)
0809 04:50:27.382841 (+ 5487us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:27.382895 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:27.384937 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:27.385572 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:27.385582 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:27.386209 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:27.386936 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:27.386959 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:27.388181 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:27.388185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:27.389346 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:27.389358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:27.390096 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:27.390103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:27.390404 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:27.391011 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:27.391037 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:27.393375 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:27.396097 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:27.396105 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:27.396120 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:27.396511 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:27.396856 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:27.396861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:27.396865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:27.397032 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:27.397524 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:27.397530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:27.397819 (+ 289us) client_negotiation.cc:769] Sending connection context
0809 04:50:27.398036 (+ 217us) client_negotiation.cc:240] Negotiation successful
0809 04:50:27.398545 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5205,"thread_start_us":4983,"threads_started":1}
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
I20260809 04:50:27.401062 9178 heartbeater.cc:347] Connected to a master server at 127.31.93.62:38019
I20260809 04:50:27.401327 9178 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:27.402032 9178 heartbeater.cc:511] Master 127.31.93.62:38019 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
I20260809 04:50:27.404167 8868 ts_manager.cc:194] Registered new tserver with Master: a194bff1512e4e3f98186db10c080b28 (127.31.93.1:44813)
I20260809 04:50:27.405633 8868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:43979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:50:27.424666 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:38019
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:27.728242 9184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:27.728488 9184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:27.728626 9184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:27.735416 9184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:27.735505 9184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:27.735651 9184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:50:27.744635 9184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:38019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:27.745836 9184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:27.747133 9184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:27.759315 9198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:27.759292 9199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:27.759665 9203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:27.760319 9184 server_base.cc:1034] running on GCE node
I20260809 04:50:27.760885 9184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:27.761801 9184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:27.763005 9184 hybrid_clock.cc:648] HybridClock initialized: now 1786251027762953 us; error 43 us; skew 500 ppm
I20260809 04:50:27.763300 9184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:27 dist-test-slave-4tdj krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251027, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:27.766283 9184 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:50:27.775389 9184 webserver.cc:460] Webserver started at http://127.31.93.2:34235/ using document root <none> and password file <none>
I20260809 04:50:27.776036 9184 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:27.776144 9184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:27.776433 9184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:27.779286 9184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "24c22a48d4df4c6785f1e70d17058f16"
format_stamp: "Formatted at 2026-08-09 04:50:27 on dist-test-slave-4tdj"
server_key: "f5849640462b23a143ad15f1a7d4f74c"
server_key_iv: "745adf48e9ba976acce47831cbbaeb79"
server_key_version: "encryptionkey@0"
I20260809 04:50:27.780058 9184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "24c22a48d4df4c6785f1e70d17058f16"
format_stamp: "Formatted at 2026-08-09 04:50:27 on dist-test-slave-4tdj"
server_key: "f5849640462b23a143ad15f1a7d4f74c"
server_key_iv: "745adf48e9ba976acce47831cbbaeb79"
server_key_version: "encryptionkey@0"
I20260809 04:50:27.784945 9184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 04:50:27.788178 9210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:27.789336 9184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:50:27.789480 9184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "24c22a48d4df4c6785f1e70d17058f16"
format_stamp: "Formatted at 2026-08-09 04:50:27 on dist-test-slave-4tdj"
server_key: "f5849640462b23a143ad15f1a7d4f74c"
server_key_iv: "745adf48e9ba976acce47831cbbaeb79"
server_key_version: "encryptionkey@0"
I20260809 04:50:27.789659 9184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:27.805089 9184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:27.809122 9184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:27.809367 9184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:27.810240 9184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:27.811484 9184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:27.811564 9184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:27.811682 9184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:27.811733 9184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:27.841460 9184 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:36643
I20260809 04:50:27.841504 9392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:36643 every 8 connection(s)
I20260809 04:50:27.842679 9184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:50:27.851326 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 9184
I20260809 04:50:27.851504 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:50:27.852041 32116 external_mini_cluster.cc:1428] Setting key dfaebc6a6c01098b69873fdb8dfedd66
Aug 09 04:50:27 dist-test-slave-4tdj krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251027, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:27.861646 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:27.845001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:51007 (local address 127.31.93.62:38019)
0809 04:50:27.845184 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:27.845189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:27.846256 (+ 1067us) server_negotiation.cc:407] Connection header received
0809 04:50:27.848008 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:27.848013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:27.848104 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:27.848193 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:27.849593 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:27.850328 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:27.852277 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:27.852709 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:27.852835 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:27.856294 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:27.856299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:27.856302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:27.856334 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:27.858888 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:27.859657 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:27.859660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:27.859663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:27.859781 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:27.860197 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:27.860200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:27.860202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:27.860434 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:27.860533 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:27.861252 (+ 719us) server_negotiation.cc:299] Negotiation successful
0809 04:50:27.861409 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 04:50:27.862763 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:27.845403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:38019 (local address 127.31.93.2:51007)
0809 04:50:27.846103 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:27.846140 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:27.847751 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:27.848425 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:27.848433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:27.848904 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:27.849430 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:27.849446 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:27.850473 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:27.850477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:27.852131 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:27.852146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:27.852938 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:27.852946 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:27.853171 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:27.853760 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:27.853780 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:27.856074 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:27.859016 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:27.859024 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:27.859039 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:27.859550 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:27.859872 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:27.859876 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:27.859879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:27.860106 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:27.860541 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:27.860547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:27.860916 (+ 369us) client_negotiation.cc:769] Sending connection context
0809 04:50:27.861189 (+ 273us) client_negotiation.cc:240] Negotiation successful
0809 04:50:27.861653 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":269,"threads_started":1}
I20260809 04:50:27.864157 9393 heartbeater.cc:347] Connected to a master server at 127.31.93.62:38019
I20260809 04:50:27.864419 9393 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:27.865073 9393 heartbeater.cc:511] Master 127.31.93.62:38019 requested a full tablet report, sending...
I20260809 04:50:27.866463 8868 ts_manager.cc:194] Registered new tserver with Master: 24c22a48d4df4c6785f1e70d17058f16 (127.31.93.2:36643)
I20260809 04:50:27.867246 8868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:51007
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:50:27.899271 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:38019
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:28.210491 9409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:28.210764 9409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:28.210909 9409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:28.217845 9409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:28.217954 9409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:28.218111 9409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:50:28.227378 9409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:38019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:28.228539 9409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:28.229771 9409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:28.242148 9417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:28.242067 9419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:28.242090 9421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:28.242805 9409 server_base.cc:1034] running on GCE node
I20260809 04:50:28.243359 9409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:28.244226 9409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:28.245440 9409 hybrid_clock.cc:648] HybridClock initialized: now 1786251028245388 us; error 36 us; skew 500 ppm
I20260809 04:50:28.245741 9409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251028, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:28.251832 9409 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:50:28.253183 9409 webserver.cc:460] Webserver started at http://127.31.93.3:41455/ using document root <none> and password file <none>
I20260809 04:50:28.253820 9409 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:28.253935 9409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:28.254208 9409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:28.256968 9409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "fe345f999b7f4e9f894179a020c2c5ff"
format_stamp: "Formatted at 2026-08-09 04:50:28 on dist-test-slave-4tdj"
server_key: "e7236870b95021a36716b0057713e9f3"
server_key_iv: "84cccc4f982f736b751d17ec0fb9b79a"
server_key_version: "encryptionkey@0"
I20260809 04:50:28.257704 9409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "fe345f999b7f4e9f894179a020c2c5ff"
format_stamp: "Formatted at 2026-08-09 04:50:28 on dist-test-slave-4tdj"
server_key: "e7236870b95021a36716b0057713e9f3"
server_key_iv: "84cccc4f982f736b751d17ec0fb9b79a"
server_key_version: "encryptionkey@0"
I20260809 04:50:28.262465 9409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260809 04:50:28.265640 9432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:28.266863 9409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:50:28.267014 9409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "fe345f999b7f4e9f894179a020c2c5ff"
format_stamp: "Formatted at 2026-08-09 04:50:28 on dist-test-slave-4tdj"
server_key: "e7236870b95021a36716b0057713e9f3"
server_key_iv: "84cccc4f982f736b751d17ec0fb9b79a"
server_key_version: "encryptionkey@0"
I20260809 04:50:28.267195 9409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:28.279865 9409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:28.283334 9409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:28.283591 9409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:28.284426 9409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:28.285704 9409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:28.285781 9409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:28.285902 9409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:28.285971 9409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:28.317422 9409 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:42605
I20260809 04:50:28.317459 9617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:42605 every 8 connection(s)
I20260809 04:50:28.318665 9409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:50:28.326779 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 9409
I20260809 04:50:28.326936 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:50:28.327374 32116 external_mini_cluster.cc:1428] Setting key cd09425a937a0b894d3c9a2f5d39c3d9
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251028, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:28.336934 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:28.321106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:42593 (local address 127.31.93.62:38019)
0809 04:50:28.321273 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:28.321278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:28.322425 (+ 1147us) server_negotiation.cc:407] Connection header received
0809 04:50:28.324177 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:28.324182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:28.324294 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:28.324384 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:28.325839 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:28.326637 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:28.328321 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:28.328793 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:28.328927 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:28.332266 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:28.332270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:28.332273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:28.332305 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:28.334603 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:28.335213 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:28.335216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:28.335218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:28.335327 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:28.335645 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:28.335648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:28.335657 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:28.335901 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:28.336004 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:28.336543 (+ 539us) server_negotiation.cc:299] Negotiation successful
0809 04:50:28.336701 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:50:28.337813 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:28.321546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:38019 (local address 127.31.93.3:42593)
0809 04:50:28.322242 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:28.322285 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:28.323916 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:28.324621 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:28.324629 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:28.325105 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:28.325674 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:28.325691 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:28.326781 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:28.326785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:28.328185 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:28.328198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:28.328994 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:28.329000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:28.329284 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:28.329887 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:28.329909 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:28.332064 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:28.334719 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:28.334725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:28.334734 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:28.335128 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:28.335396 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:28.335399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:28.335401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:28.335571 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:28.335985 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:28.335990 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:28.336265 (+ 275us) client_negotiation.cc:769] Sending connection context
0809 04:50:28.336472 (+ 207us) client_negotiation.cc:240] Negotiation successful
0809 04:50:28.336813 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":269,"threads_started":1}
I20260809 04:50:28.339016 9618 heartbeater.cc:347] Connected to a master server at 127.31.93.62:38019
I20260809 04:50:28.339291 9618 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:28.339929 9618 heartbeater.cc:511] Master 127.31.93.62:38019 requested a full tablet report, sending...
I20260809 04:50:28.341357 8868 ts_manager.cc:194] Registered new tserver with Master: fe345f999b7f4e9f894179a020c2c5ff (127.31.93.3:42605)
I20260809 04:50:28.342343 8868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:42593
I20260809 04:50:28.342612 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/test-admin.keytab.
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:28.370255 32116 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260809 04:50:28.375123 32116 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/krb5cc' of type FILE
W20260809 04:50:28.375177 9629 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:28.372712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55838 (local address 127.31.93.1:44813)
0809 04:50:28.373430 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:28.373439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:28.373472 (+ 33us) server_negotiation.cc:407] Connection header received
0809 04:50:28.373545 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:28.373552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:28.373811 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:28.373973 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:28.374753 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55838: BlockingRecv error: recv got EOF from 127.0.0.1:55838 (error 108)
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":422,"threads_started":1}
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:28.376677 32116 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:28.378158 32116 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:50:28.391724 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:28.379900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55840 (local address 127.31.93.1:44813)
0809 04:50:28.380039 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:28.380043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:28.380324 (+ 281us) server_negotiation.cc:407] Connection header received
0809 04:50:28.380444 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:28.380447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:28.380496 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:28.380566 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:28.381439 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:28.382399 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:28.383516 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:28.384012 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:28.384102 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:28.387258 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:28.387269 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:28.387280 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:28.387308 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:28.389619 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:28.390147 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:28.390160 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:28.390162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:28.390268 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:28.390581 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:28.390584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:28.390586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:28.390890 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:28.391088 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:28.391315 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 04:50:28.391492 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260809 04:50:28.396807 9629 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:28.395420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55844 (local address 127.31.93.1:44813)
0809 04:50:28.395841 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:28.395845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:28.395860 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:50:28.395964 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:28.395968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:28.396013 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:28.396081 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:28.396591 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55844: BlockingRecv error: recv got EOF from 127.0.0.1:55844 (error 108)
Metrics: {"server-negotiator.queue_time_us":300}
W20260809 04:50:28.397819 32116 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/krb5cc)
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:28.399112 32116 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:28.400233 32116 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:50:28.408486 9178 heartbeater.cc:503] Master 127.31.93.62:38019 was elected leader, sending a full tablet report...
I20260809 04:50:28.412628 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:28.401242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.31.93.1:44813)
0809 04:50:28.401456 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:28.401462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:28.401723 (+ 261us) server_negotiation.cc:407] Connection header received
0809 04:50:28.401818 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:28.401821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:28.401867 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:28.401934 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:28.402847 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:28.403524 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:28.404643 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:28.405076 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:28.405161 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:28.408289 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:28.408293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:28.408296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:28.408341 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:28.410801 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:28.411302 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:28.411305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:28.411308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:28.411410 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:28.411685 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:28.411688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:28.411690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:28.411903 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:28.412084 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:28.412249 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 04:50:28.412380 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260809 04:50:28.415992 9629 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:28.414652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55860 (local address 127.31.93.1:44813)
0809 04:50:28.414780 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:28.414784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:28.415072 (+ 288us) server_negotiation.cc:407] Connection header received
0809 04:50:28.415185 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:28.415188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:28.415245 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:28.415315 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:28.415818 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55860: BlockingRecv error: recv got EOF from 127.0.0.1:55860 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260809 04:50:28.416286 32116 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786250977734564-32116-0/krb5kdc/krb5cc' of type FILE
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:28.417481 32116 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[8793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:28.418677 32116 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:50:28.430552 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:28.419716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.31.93.1:44813)
0809 04:50:28.419844 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:28.419849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:28.420231 (+ 382us) server_negotiation.cc:407] Connection header received
0809 04:50:28.420320 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:28.420323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:28.420369 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:28.420437 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:28.421328 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:28.421965 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:28.423099 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:28.423488 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:28.423572 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:28.426474 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:28.426478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:28.426481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:28.426507 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:28.428605 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:28.429121 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:28.429124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:28.429126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:28.429235 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:28.429492 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:28.429495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:28.429497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:28.429724 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:28.429934 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:28.430091 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 04:50:28.430247 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 04:50:28.431741 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 8968
I20260809 04:50:28.441965 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:28.432324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.31.93.1:44813)
0809 04:50:28.432444 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:28.432449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:28.432756 (+ 307us) server_negotiation.cc:407] Connection header received
0809 04:50:28.432879 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:28.432884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:28.432936 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:28.433009 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:28.433994 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:28.434681 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:28.435722 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:28.436108 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:28.436187 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:28.438015 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:28.438019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:28.438022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:28.438048 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:28.440089 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:28.440601 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:28.440605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:28.440607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:28.440729 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:28.440976 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:28.440979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:28.440981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:28.441186 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:28.441351 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:28.441538 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 04:50:28.441681 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 04:50:28.442448 9170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:28.522387 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 8968
I20260809 04:50:28.541625 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 9184
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:50:28.555840 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:28.542546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35762 (local address 127.31.93.2:36643)
0809 04:50:28.543064 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:28.543072 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:28.543097 (+ 25us) server_negotiation.cc:407] Connection header received
0809 04:50:28.543155 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:28.543161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:28.543376 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:28.543504 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:28.544375 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:28.545396 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:28.546656 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:28.547192 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:28.547286 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:28.550772 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:28.550782 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:28.550795 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:28.550827 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:28.553459 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:28.554036 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:28.554043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:28.554045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:28.554158 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:28.554474 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:28.554477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:28.554479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:28.554814 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:28.555115 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:28.555296 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 04:50:28.555546 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":255,"threads_started":1}
I20260809 04:50:28.557058 9386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:28.636509 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 9184
I20260809 04:50:28.653837 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 9409
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[8793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:50:28.668591 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:28.654730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.31.93.3:42605)
0809 04:50:28.655326 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:28.655332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:28.655361 (+ 29us) server_negotiation.cc:407] Connection header received
0809 04:50:28.655420 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:28.655427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:28.655644 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:28.655785 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:28.656868 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:28.657923 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:28.659235 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:28.659808 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:28.659921 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:28.663619 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:28.663631 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:28.663646 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:28.663687 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:28.666381 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:28.666966 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:28.666973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:28.666975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:28.667092 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:28.667368 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:28.667371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:28.667373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:28.667705 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:28.667933 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:28.668102 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 04:50:28.668311 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":270,"threads_started":1}
I20260809 04:50:28.669829 9613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:28.750605 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 9409
I20260809 04:50:28.768249 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 8816
I20260809 04:50:28.769373 8917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:28.870535 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 8816
2026-08-09T04:50:28Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3911 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[9655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[9655](info): set up 2 sockets
Aug 09 04:50:28 dist-test-slave-4tdj krb5kdc[9655](info): commencing operation
krb5kdc: starting...
W20260809 04:50:30.083983 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:50:30 dist-test-slave-4tdj krb5kdc[9655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:50:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.62.keytab.
Entry for principal oryx/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.62.keytab.
I20260809 04:50:30.295104 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:45495
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.62.keytab
--principal=oryx/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:45495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:30.608587 9676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:30.608879 9676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:30.609017 9676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:30.616024 9676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:30.616137 9676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:30.616226 9676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:30.616307 9676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:30.616369 9676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:50:30.625839 9676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:45495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.62.keytab
--principal=oryx/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:45495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:30.627128 9676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:30.628470 9676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:30.639560 9684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:30.639716 9685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:30.639597 9688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:30.640532 9676 server_base.cc:1034] running on GCE node
I20260809 04:50:30.641249 9676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:30.642599 9676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:30.643822 9676 hybrid_clock.cc:648] HybridClock initialized: now 1786251030643768 us; error 43 us; skew 500 ppm
I20260809 04:50:30.644151 9676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:30 dist-test-slave-4tdj krb5kdc[9655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251030, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:30.647042 9676 init.cc:377] Logged in from keytab as oryx/127.31.93.62@KRBTEST.COM (short username oryx)
I20260809 04:50:30.648362 9676 webserver.cc:460] Webserver started at http://127.31.93.62:41853/ using document root <none> and password file <none>
I20260809 04:50:30.648995 9676 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:30.649108 9676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:30.649392 9676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:30.652264 9676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "391a359744cb4913b6a7a3fdfaff318a"
format_stamp: "Formatted at 2026-08-09 04:50:30 on dist-test-slave-4tdj"
server_key: "a8b367da510ae3b1703997cfccfc6bc1"
server_key_iv: "4dea840324040b4e78e1704fe279f259"
server_key_version: "encryptionkey@0"
I20260809 04:50:30.653003 9676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "391a359744cb4913b6a7a3fdfaff318a"
format_stamp: "Formatted at 2026-08-09 04:50:30 on dist-test-slave-4tdj"
server_key: "a8b367da510ae3b1703997cfccfc6bc1"
server_key_iv: "4dea840324040b4e78e1704fe279f259"
server_key_version: "encryptionkey@0"
I20260809 04:50:30.657898 9676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260809 04:50:30.661242 9702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:30.662478 9676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:50:30.662622 9676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "391a359744cb4913b6a7a3fdfaff318a"
format_stamp: "Formatted at 2026-08-09 04:50:30 on dist-test-slave-4tdj"
server_key: "a8b367da510ae3b1703997cfccfc6bc1"
server_key_iv: "4dea840324040b4e78e1704fe279f259"
server_key_version: "encryptionkey@0"
I20260809 04:50:30.662792 9676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:30.686574 9676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:30.690189 9676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:30.690464 9676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:30.706907 9676 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:45495
I20260809 04:50:30.706952 9776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:45495 every 8 connection(s)
I20260809 04:50:30.708170 9676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:50:30.711342 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 9676
I20260809 04:50:30.711557 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:50:30.711985 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:30.712442 32116 external_mini_cluster.cc:1428] Setting key 82994df07b20c99b5a13bde5e6d641eb
I20260809 04:50:30.725847 9778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a: Bootstrap starting.
Aug 09 04:50:30 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM
I20260809 04:50:30.729301 9778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:30.730382 9778 log.cc:824] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:30.732959 9778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a: No bootstrap required, opened a new log
I20260809 04:50:30.734531 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:30.714503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.31.93.62:45495)
0809 04:50:30.715292 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:30.715304 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:30.715346 (+ 42us) server_negotiation.cc:407] Connection header received
0809 04:50:30.716901 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:30.716921 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:30.717304 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:30.717627 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:30.719435 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:30.720672 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:30.722371 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:30.723037 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:30.723160 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:30.727597 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:30.727611 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:30.727629 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:30.727679 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:30.730835 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:30.731465 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:30.731472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:30.731477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:30.731647 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:30.731990 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:30.731994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:30.731996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:30.732472 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:30.732762 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:30.733170 (+ 408us) server_negotiation.cc:299] Negotiation successful
0809 04:50:30.733559 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":366,"threads_started":1}
I20260809 04:50:30.740332 9778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } }
I20260809 04:50:30.740758 9778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:30.740880 9778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391a359744cb4913b6a7a3fdfaff318a, State: Initialized, Role: FOLLOWER
I20260809 04:50:30.741466 9778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } }
I20260809 04:50:30.741704 9778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:50:30.741809 9778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:50:30.741920 9778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:30.743911 9778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } }
I20260809 04:50:30.744325 9778 leader_election.cc:302] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 391a359744cb4913b6a7a3fdfaff318a; no voters:
I20260809 04:50:30.744828 9778 leader_election.cc:288] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:50:30.744946 9784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:30.745579 9784 raft_consensus.cc:686] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [term 1 LEADER]: Becoming Leader. State: Replica: 391a359744cb4913b6a7a3fdfaff318a, State: Running, Role: LEADER
I20260809 04:50:30.746145 9784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } }
I20260809 04:50:30.746902 9778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:30.748886 9788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 391a359744cb4913b6a7a3fdfaff318a. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } }
I20260809 04:50:30.749197 9788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:30.748970 9786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } }
I20260809 04:50:30.749521 9786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:30.754876 9793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:50:30.759382 9793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:50:30.767623 9793 catalog_manager.cc:1347] Generated new cluster ID: bbc24b55091f4398b1e1f66b7eb68697
I20260809 04:50:30.767729 9793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:50:30.777606 9793 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:50:30.779385 9793 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:50:30.792588 9793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a: Generated new TSK 0
I20260809 04:50:30.793545 9793 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.1.keytab.
Entry for principal oryx/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.1.keytab.
I20260809 04:50:30.841904 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:45495
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.1.keytab
--principal=oryx/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:31.187394 9814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:31.187652 9814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:31.187752 9814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:31.194692 9814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:31.194784 9814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:31.194900 9814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:50:31.203985 9814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.1.keytab
--principal=oryx/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:45495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:31.205106 9814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:31.206444 9814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:31.218552 9823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:31.218609 9819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:31.218614 9820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:31.219305 9814 server_base.cc:1034] running on GCE node
I20260809 04:50:31.219863 9814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:31.220700 9814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:31.221895 9814 hybrid_clock.cc:648] HybridClock initialized: now 1786251031221857 us; error 28 us; skew 500 ppm
I20260809 04:50:31.222179 9814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:31 dist-test-slave-4tdj krb5kdc[9655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251031, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:31.225152 9814 init.cc:377] Logged in from keytab as oryx/127.31.93.1@KRBTEST.COM (short username oryx)
I20260809 04:50:31.226466 9814 webserver.cc:460] Webserver started at http://127.31.93.1:46327/ using document root <none> and password file <none>
I20260809 04:50:31.227079 9814 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:31.227191 9814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:31.227466 9814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:31.230402 9814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "67bceba2f21444d69fdc7d07a7242c3c"
format_stamp: "Formatted at 2026-08-09 04:50:31 on dist-test-slave-4tdj"
server_key: "35d36a65af47c9313a7f668571cf150e"
server_key_iv: "e81664379b2c222d7a107d6b95c4e69f"
server_key_version: "encryptionkey@0"
I20260809 04:50:31.231242 9814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "67bceba2f21444d69fdc7d07a7242c3c"
format_stamp: "Formatted at 2026-08-09 04:50:31 on dist-test-slave-4tdj"
server_key: "35d36a65af47c9313a7f668571cf150e"
server_key_iv: "e81664379b2c222d7a107d6b95c4e69f"
server_key_version: "encryptionkey@0"
I20260809 04:50:31.236363 9814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260809 04:50:31.239564 9833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:31.240749 9814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:50:31.240895 9814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "67bceba2f21444d69fdc7d07a7242c3c"
format_stamp: "Formatted at 2026-08-09 04:50:31 on dist-test-slave-4tdj"
server_key: "35d36a65af47c9313a7f668571cf150e"
server_key_iv: "e81664379b2c222d7a107d6b95c4e69f"
server_key_version: "encryptionkey@0"
I20260809 04:50:31.241073 9814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:31.264663 9814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:31.268358 9814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:31.268610 9814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:31.269459 9814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:31.270712 9814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:31.270782 9814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:31.270893 9814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:31.270954 9814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:31.300134 9814 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:33309
I20260809 04:50:31.300196 10010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:33309 every 8 connection(s)
I20260809 04:50:31.301326 9814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:50:31.304411 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 9814
I20260809 04:50:31.304594 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:50:31.305279 32116 external_mini_cluster.cc:1428] Setting key 1ff9404f856de31b10554caf5be53f24
Aug 09 04:50:31 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251031, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM
I20260809 04:50:31.320072 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:31.303611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:44559 (local address 127.31.93.62:45495)
0809 04:50:31.304034 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:31.304039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:31.304946 (+ 907us) server_negotiation.cc:407] Connection header received
0809 04:50:31.307121 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:31.307125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:31.307201 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:31.307292 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:31.308680 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:31.309446 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:31.310980 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:31.311384 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:31.311495 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:31.314743 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:31.314748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:31.314751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:31.314788 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:31.317230 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:31.317894 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:31.317900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:31.317904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:31.318022 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:31.318461 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:31.318466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:31.318470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:31.318720 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:31.318852 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:31.319505 (+ 653us) server_negotiation.cc:299] Negotiation successful
0809 04:50:31.319719 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":220,"threads_started":1}
I20260809 04:50:31.320914 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:31.304011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.1:44559)
0809 04:50:31.304772 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:31.304822 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:31.306747 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:31.307528 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:31.307536 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:31.308000 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:31.308515 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:31.308532 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:31.309601 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:31.309604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:31.310838 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:31.310851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:31.311608 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:31.311616 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:31.311822 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:31.312424 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:31.312444 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:31.314520 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:31.317356 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:31.317365 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:31.317379 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:31.317777 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:31.318130 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:31.318135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:31.318139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:31.318353 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:31.318843 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:31.318850 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:31.319149 (+ 299us) client_negotiation.cc:769] Sending connection context
0809 04:50:31.319365 (+ 216us) client_negotiation.cc:240] Negotiation successful
0809 04:50:31.319846 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":306,"threads_started":1}
I20260809 04:50:31.322327 10011 heartbeater.cc:347] Connected to a master server at 127.31.93.62:45495
I20260809 04:50:31.322654 10011 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:31.323460 10011 heartbeater.cc:511] Master 127.31.93.62:45495 requested a full tablet report, sending...
I20260809 04:50:31.325541 9725 ts_manager.cc:194] Registered new tserver with Master: 67bceba2f21444d69fdc7d07a7242c3c (127.31.93.1:33309)
I20260809 04:50:31.326978 9725 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:44559
WARNING: no policy specified for oryx/127.31.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.2.keytab.
Entry for principal oryx/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.2.keytab.
I20260809 04:50:31.353870 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:45495
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.2.keytab
--principal=oryx/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:31.662071 10023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:31.662361 10023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:31.662497 10023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:31.669422 10023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:31.669528 10023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:31.669678 10023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:50:31.678808 10023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.2.keytab
--principal=oryx/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:45495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:31.679955 10023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:31.681223 10023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:31.693361 10032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:31.693375 10031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:31.693379 10038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:31.694450 10023 server_base.cc:1034] running on GCE node
I20260809 04:50:31.695020 10023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:31.695876 10023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:31.697072 10023 hybrid_clock.cc:648] HybridClock initialized: now 1786251031697031 us; error 27 us; skew 500 ppm
I20260809 04:50:31.697356 10023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:31 dist-test-slave-4tdj krb5kdc[9655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251031, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:31.700209 10023 init.cc:377] Logged in from keytab as oryx/127.31.93.2@KRBTEST.COM (short username oryx)
I20260809 04:50:31.701525 10023 webserver.cc:460] Webserver started at http://127.31.93.2:35897/ using document root <none> and password file <none>
I20260809 04:50:31.702137 10023 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:31.702255 10023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:31.702548 10023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:31.705255 10023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "3480182938594d0a84d8c3986d8ee6be"
format_stamp: "Formatted at 2026-08-09 04:50:31 on dist-test-slave-4tdj"
server_key: "b27569ad7ca0bf84f8cf841cb1b9e3bd"
server_key_iv: "2438b0547b0c37c818dd694bbdb1699d"
server_key_version: "encryptionkey@0"
I20260809 04:50:31.705988 10023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "3480182938594d0a84d8c3986d8ee6be"
format_stamp: "Formatted at 2026-08-09 04:50:31 on dist-test-slave-4tdj"
server_key: "b27569ad7ca0bf84f8cf841cb1b9e3bd"
server_key_iv: "2438b0547b0c37c818dd694bbdb1699d"
server_key_version: "encryptionkey@0"
I20260809 04:50:31.710927 10023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260809 04:50:31.714200 10050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:31.715544 10023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:50:31.715685 10023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "3480182938594d0a84d8c3986d8ee6be"
format_stamp: "Formatted at 2026-08-09 04:50:31 on dist-test-slave-4tdj"
server_key: "b27569ad7ca0bf84f8cf841cb1b9e3bd"
server_key_iv: "2438b0547b0c37c818dd694bbdb1699d"
server_key_version: "encryptionkey@0"
I20260809 04:50:31.715860 10023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:31.728108 10023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:31.731611 10023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:31.731858 10023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:31.732695 10023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:31.733938 10023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:31.734015 10023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:31.734138 10023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:31.734196 10023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:31.763842 10023 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:40459
I20260809 04:50:31.763885 10222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:40459 every 8 connection(s)
I20260809 04:50:31.765015 10023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:50:31.770208 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 10023
I20260809 04:50:31.770394 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:50:31.770937 32116 external_mini_cluster.cc:1428] Setting key 985f4387568a95aed2e5ae369b93c997
Aug 09 04:50:31 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251031, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM
I20260809 04:50:31.783689 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:31.767486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:48083 (local address 127.31.93.62:45495)
0809 04:50:31.767655 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:31.767659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:31.768712 (+ 1053us) server_negotiation.cc:407] Connection header received
0809 04:50:31.770471 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:31.770477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:31.770569 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:31.770686 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:31.772165 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:31.772999 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:31.774474 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:31.774870 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:31.774978 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:31.778436 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:31.778441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:31.778445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:31.778477 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:31.780875 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:31.781530 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:31.781536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:31.781539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:31.781696 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:31.782065 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:31.782070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:31.782073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:31.782423 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:31.782559 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:31.783167 (+ 608us) server_negotiation.cc:299] Negotiation successful
0809 04:50:31.783381 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 04:50:31.784350 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:31.767891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.2:48083)
0809 04:50:31.768557 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:31.768608 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:31.770166 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:31.770854 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:31.770862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:31.771358 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:31.771997 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:31.772014 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:31.773135 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:31.773139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:31.774350 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:31.774363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:31.775076 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:31.775084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:31.775292 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:31.775864 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:31.775886 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:31.778213 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:31.781014 (+ 2801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:31.781020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:31.781030 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:31.781415 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:31.781785 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:31.781788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:31.781790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:31.781964 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:31.782524 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:31.782530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:31.782812 (+ 282us) client_negotiation.cc:769] Sending connection context
0809 04:50:31.783016 (+ 204us) client_negotiation.cc:240] Negotiation successful
0809 04:50:31.783361 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":255,"threads_started":1}
I20260809 04:50:31.785615 10223 heartbeater.cc:347] Connected to a master server at 127.31.93.62:45495
I20260809 04:50:31.785923 10223 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:31.786768 10223 heartbeater.cc:511] Master 127.31.93.62:45495 requested a full tablet report, sending...
I20260809 04:50:31.788283 9725 ts_manager.cc:194] Registered new tserver with Master: 3480182938594d0a84d8c3986d8ee6be (127.31.93.2:40459)
I20260809 04:50:31.789043 9725 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:48083
WARNING: no policy specified for oryx/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.3.keytab.
Entry for principal oryx/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.3.keytab.
I20260809 04:50:31.819043 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:45495
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.3.keytab
--principal=oryx/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:32.138618 10233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:32.138900 10233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:32.139052 10233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:32.146442 10233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:32.146567 10233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:32.146733 10233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:50:32.156306 10233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.3.keytab
--principal=oryx/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:45495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:32.157728 10233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:32.159083 10233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:32.171691 10251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:32.171674 10259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:32.171742 10253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:32.172811 10233 server_base.cc:1034] running on GCE node
I20260809 04:50:32.173352 10233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:32.174242 10233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:32.175472 10233 hybrid_clock.cc:648] HybridClock initialized: now 1786251032175406 us; error 49 us; skew 500 ppm
I20260809 04:50:32.175743 10233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:32 dist-test-slave-4tdj krb5kdc[9655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251032, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:32.178741 10233 init.cc:377] Logged in from keytab as oryx/127.31.93.3@KRBTEST.COM (short username oryx)
I20260809 04:50:32.180063 10233 webserver.cc:460] Webserver started at http://127.31.93.3:32855/ using document root <none> and password file <none>
I20260809 04:50:32.180687 10233 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:32.180787 10233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:32.181030 10233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:32.183861 10233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "c5ecb8b991eb4fceae5fb19e581391ee"
format_stamp: "Formatted at 2026-08-09 04:50:32 on dist-test-slave-4tdj"
server_key: "fae77142420c4212f3751eb826a02c39"
server_key_iv: "7a15a382ceeaf33daa71770bca4d4232"
server_key_version: "encryptionkey@0"
I20260809 04:50:32.184634 10233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "c5ecb8b991eb4fceae5fb19e581391ee"
format_stamp: "Formatted at 2026-08-09 04:50:32 on dist-test-slave-4tdj"
server_key: "fae77142420c4212f3751eb826a02c39"
server_key_iv: "7a15a382ceeaf33daa71770bca4d4232"
server_key_version: "encryptionkey@0"
I20260809 04:50:32.189543 10233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 04:50:32.192791 10268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:32.194065 10233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:50:32.194211 10233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "c5ecb8b991eb4fceae5fb19e581391ee"
format_stamp: "Formatted at 2026-08-09 04:50:32 on dist-test-slave-4tdj"
server_key: "fae77142420c4212f3751eb826a02c39"
server_key_iv: "7a15a382ceeaf33daa71770bca4d4232"
server_key_version: "encryptionkey@0"
I20260809 04:50:32.194420 10233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:32.212056 10233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:32.215975 10233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:32.216212 10233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:32.217046 10233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:32.218364 10233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:32.218441 10233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:32.218534 10233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:32.218578 10233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:32.248636 10233 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:43813
I20260809 04:50:32.248716 10453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:43813 every 8 connection(s)
I20260809 04:50:32.249810 10233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:50:32.255473 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 10233
I20260809 04:50:32.255647 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:50:32.256093 32116 external_mini_cluster.cc:1428] Setting key d0cd5b6868266838d95f34920c8a0613
Aug 09 04:50:32 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251032, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM
I20260809 04:50:32.268370 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:32.252181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:47441 (local address 127.31.93.62:45495)
0809 04:50:32.252348 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:32.252352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:32.253450 (+ 1098us) server_negotiation.cc:407] Connection header received
0809 04:50:32.255230 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:32.255236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:32.255314 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:32.255399 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:32.257022 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.257904 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.259490 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.259876 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.259981 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:32.263168 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:32.263173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:32.263176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:32.263209 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:32.265518 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:32.266139 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:32.266143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:32.266145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:32.266265 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:32.266715 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:32.266720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:32.266723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:32.267041 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:32.267170 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:32.267803 (+ 633us) server_negotiation.cc:299] Negotiation successful
0809 04:50:32.268044 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:50:32.269131 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:32.252593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.3:47441)
0809 04:50:32.253292 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:32.253328 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:32.254961 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:32.255663 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:32.255672 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:32.256174 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:32.256834 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:32.256859 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.258045 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.258051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:32.259361 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:32.259374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.260087 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.260094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:32.260301 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:32.260839 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:32.260859 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:32.262982 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:32.265632 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:32.265638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:32.265648 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:32.266046 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:32.266395 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:32.266400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:32.266404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:32.266599 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:32.267156 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:32.267163 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:32.267438 (+ 275us) client_negotiation.cc:769] Sending connection context
0809 04:50:32.267671 (+ 233us) client_negotiation.cc:240] Negotiation successful
0809 04:50:32.268111 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":270,"threads_started":1}
I20260809 04:50:32.270419 10454 heartbeater.cc:347] Connected to a master server at 127.31.93.62:45495
I20260809 04:50:32.270725 10454 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:32.271461 10454 heartbeater.cc:511] Master 127.31.93.62:45495 requested a full tablet report, sending...
I20260809 04:50:32.272738 9724 ts_manager.cc:194] Registered new tserver with Master: c5ecb8b991eb4fceae5fb19e581391ee (127.31.93.3:43813)
I20260809 04:50:32.273527 9724 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:47441
I20260809 04:50:32.283072 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.93.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.93.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.93.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.61.keytab.
Entry for principal oryx/127.31.93.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.61.keytab.
I20260809 04:50:32.329977 10011 heartbeater.cc:503] Master 127.31.93.62:45495 was elected leader, sending a full tablet report...
I20260809 04:50:32.331190 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.61:41063
--webserver_interface=127.31.93.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.61.keytab
--principal=oryx/127.31.93.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:45495,127.31.93.61:41063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:32.643538 10460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:32.643793 10460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:32.643878 10460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:32.650902 10460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:32.650997 10460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:32.651048 10460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:32.651103 10460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:32.651145 10460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:50:32.660418 10460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:45495,127.31.93.61:41063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.61.keytab
--principal=oryx/127.31.93.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.61:41063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.93.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
W20260809 04:50:32.660701 10460 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.31.93.62:45495,127.31.93.61:41063'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260809 04:50:32.661618 10460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:32.662832 10460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:32.673738 10467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:32.673779 10469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:32.673803 10471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:32.674683 10460 server_base.cc:1034] running on GCE node
I20260809 04:50:32.675381 10460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:32.676649 10460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:32.677848 10460 hybrid_clock.cc:648] HybridClock initialized: now 1786251032677788 us; error 43 us; skew 500 ppm
I20260809 04:50:32.678131 10460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:32 dist-test-slave-4tdj krb5kdc[9655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251032, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:32.681075 10460 init.cc:377] Logged in from keytab as oryx/127.31.93.61@KRBTEST.COM (short username oryx)
I20260809 04:50:32.682488 10460 webserver.cc:460] Webserver started at http://127.31.93.61:37987/ using document root <none> and password file <none>
I20260809 04:50:32.683109 10460 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:32.683223 10460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:32.683508 10460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:32.686339 10460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/data/instance:
uuid: "025ce77beaad4885a83c296e1694006e"
format_stamp: "Formatted at 2026-08-09 04:50:32 on dist-test-slave-4tdj"
server_key: "5280d6468025f890c380d8fbe321d806"
server_key_iv: "cce6c544b127496d32c273cc0fa99f16"
server_key_version: "encryptionkey@0"
I20260809 04:50:32.687096 10460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/wal/instance:
uuid: "025ce77beaad4885a83c296e1694006e"
format_stamp: "Formatted at 2026-08-09 04:50:32 on dist-test-slave-4tdj"
server_key: "5280d6468025f890c380d8fbe321d806"
server_key_iv: "cce6c544b127496d32c273cc0fa99f16"
server_key_version: "encryptionkey@0"
I20260809 04:50:32.691792 10460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:50:32.694906 10480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:32.696082 10460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:50:32.696242 10460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/wal
uuid: "025ce77beaad4885a83c296e1694006e"
format_stamp: "Formatted at 2026-08-09 04:50:32 on dist-test-slave-4tdj"
server_key: "5280d6468025f890c380d8fbe321d806"
server_key_iv: "cce6c544b127496d32c273cc0fa99f16"
server_key_version: "encryptionkey@0"
I20260809 04:50:32.696424 10460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:32.711261 10460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:32.716029 10460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:32.716282 10460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:32.733716 10460 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.61:41063
I20260809 04:50:32.733740 10565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.61:41063 every 8 connection(s)
I20260809 04:50:32.735038 10460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/data/info.pb
I20260809 04:50:32.737721 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 10460
I20260809 04:50:32.738860 10566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:32.747824 10566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:50:32 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.93.61@KRBTEST.COM
Aug 09 04:50:32 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251032, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.61@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM
I20260809 04:50:32.761528 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:32.739974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.31.93.61:41063)
0809 04:50:32.740690 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:32.740700 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:32.740742 (+ 42us) server_negotiation.cc:407] Connection header received
0809 04:50:32.743117 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:32.743138 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:32.743488 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:32.743803 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:32.745602 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.746898 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.748584 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.749261 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.749380 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:32.754055 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:32.754062 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:32.754072 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:32.754106 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:32.757559 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:32.758170 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:32.758176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:32.758182 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:32.758364 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:32.758767 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:32.758771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:32.758775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:32.759261 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:32.759507 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:32.759918 (+ 411us) server_negotiation.cc:299] Negotiation successful
0809 04:50:32.760284 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":323,"threads_started":1}
I20260809 04:50:32.765509 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:32.750469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.0.0.1:42580)
0809 04:50:32.751138 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:32.751183 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:32.751350 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:32.751688 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:32.751695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:32.752453 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:32.752737 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:32.752747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.753817 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.753822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:32.755133 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:32.755152 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.756089 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.756097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:32.756316 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:32.756927 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:32.756951 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:32.759300 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:32.762731 (+ 3431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:32.762739 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:32.762745 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:32.763297 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:32.763792 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:32.763797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:32.763800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:32.763966 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:32.764680 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:32.764686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:32.764848 (+ 162us) client_negotiation.cc:769] Sending connection context
0809 04:50:32.765022 (+ 174us) client_negotiation.cc:240] Negotiation successful
0809 04:50:32.765241 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":257,"threads_started":1}
I20260809 04:50:32.765697 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:32.750566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.31.93.62:45495)
0809 04:50:32.750773 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:32.750779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:32.751292 (+ 513us) server_negotiation.cc:407] Connection header received
0809 04:50:32.751460 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:32.751464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:32.751527 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:32.751600 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:32.752872 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.753681 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.755278 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.755888 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.756026 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:32.759490 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:32.759496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:32.759500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:32.759540 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:32.762605 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:32.763515 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:32.763520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:32.763523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:32.763676 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:32.764093 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:32.764097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:32.764101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:32.764536 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:32.764686 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:32.765160 (+ 474us) server_negotiation.cc:299] Negotiation successful
0809 04:50:32.765382 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 04:50:32.767390 10566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:50:32 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251032, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.61@KRBTEST.COM for oryx/127.31.93.61@KRBTEST.COM
I20260809 04:50:32.778070 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:32.768098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:41063 (local address 127.0.0.1:53088)
0809 04:50:32.768273 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:32.768291 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:32.768378 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:32.768616 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:32.768619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:32.768930 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:32.769042 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:32.769049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.769939 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.769944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:32.770804 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:32.770816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.771438 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.771444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:32.771601 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:32.772043 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:32.772059 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:32.773813 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:32.776274 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:32.776278 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:32.776280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:32.776606 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:32.776893 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:32.776896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:32.776898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:32.777009 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:32.777435 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:32.777438 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:32.777562 (+ 124us) client_negotiation.cc:769] Sending connection context
0809 04:50:32.777672 (+ 110us) client_negotiation.cc:240] Negotiation successful
0809 04:50:32.777857 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 04:50:32.778277 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:32.768247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.31.93.61:41063)
0809 04:50:32.768364 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:32.768368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:32.768385 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:50:32.768458 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:32.768462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:32.768518 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:32.768587 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:32.769161 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.769809 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.770928 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.771308 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.771411 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:32.773963 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:32.773968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:32.773971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:32.773997 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:32.776162 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:32.776712 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:32.776715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:32.776718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:32.776820 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:32.777086 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:32.777089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:32.777091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:32.777346 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:32.777447 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:32.777753 (+ 306us) server_negotiation.cc:299] Negotiation successful
0809 04:50:32.777978 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":33}
I20260809 04:50:32.782514 10566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: Bootstrap starting.
I20260809 04:50:32.785346 10566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:32.786351 10566 log.cc:824] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:32.788630 10566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: No bootstrap required, opened a new log
I20260809 04:50:32.791935 10223 heartbeater.cc:503] Master 127.31.93.62:45495 was elected leader, sending a full tablet report...
I20260809 04:50:32.795732 10566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } }
I20260809 04:50:32.795996 10566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:32.796062 10566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025ce77beaad4885a83c296e1694006e, State: Initialized, Role: FOLLOWER
I20260809 04:50:32.796523 10566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } }
I20260809 04:50:32.797224 10577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } } }
I20260809 04:50:32.797511 10577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:50:32.798131 10566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: configured and running, proceeding with master startup.
W20260809 04:50:32.806075 10602 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:50:32.806200 10602 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:50:32.809249 10460 master_runner.cc:428] Detected that this master 025ce77beaad4885a83c296e1694006e is joining an existing cluster
I20260809 04:50:32.809345 10460 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.93.61:41063
I20260809 04:50:32.828820 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:32.813370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:41063 (local address 127.0.0.1:53090)
0809 04:50:32.814423 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:32.814444 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:32.814558 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:32.815004 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:32.815007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:32.815752 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:32.816100 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:32.816113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.817513 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.817517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:32.819148 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:32.819164 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.820305 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.820310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:32.820506 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:32.821216 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:32.821238 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:32.822374 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:32.826049 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:32.826053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:32.826057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:32.826608 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:32.827008 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:32.827012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:32.827015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:32.827169 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:32.827810 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:32.827814 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:32.827951 (+ 137us) client_negotiation.cc:769] Sending connection context
0809 04:50:32.828080 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 04:50:32.828368 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":76,"thread_start_us":721,"threads_started":1}
I20260809 04:50:32.829310 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:32.813198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.31.93.62:45495)
0809 04:50:32.813347 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:32.813351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:32.814108 (+ 757us) server_negotiation.cc:407] Connection header received
0809 04:50:32.814430 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:32.814435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:32.814518 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:32.814626 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:32.815945 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.817052 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.819169 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.819815 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.819910 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:32.822596 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:32.822602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:32.822607 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:32.822658 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:32.826112 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:32.826978 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:32.826984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:32.826988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:32.827138 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:32.827631 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:32.827636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:32.827639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:32.827974 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:32.828105 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:32.828739 (+ 634us) server_negotiation.cc:299] Negotiation successful
0809 04:50:32.828967 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 04:50:32.829640 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:32.813774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.31.93.61:41063)
0809 04:50:32.813964 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:32.813969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:32.814530 (+ 561us) server_negotiation.cc:407] Connection header received
0809 04:50:32.814678 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:32.814683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:32.814752 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:32.814816 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:32.816246 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.817268 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.819284 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.819916 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.820002 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:32.822528 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:32.822533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:32.822537 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:32.822567 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:32.825918 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:32.826725 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:32.826730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:32.826733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:32.826893 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:32.827270 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:32.827274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:32.827277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:32.827620 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:32.827761 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:32.829131 (+ 1370us) server_negotiation.cc:299] Negotiation successful
0809 04:50:32.829341 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 04:50:32.830703 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:32.813312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.0.0.1:42592)
0809 04:50:32.814008 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:32.814026 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:32.814215 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:32.814634 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:32.814639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:32.815350 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:32.815765 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:32.815779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.817225 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.817230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:32.819007 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:32.819021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.820222 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.820229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:32.820416 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:32.821045 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:32.821068 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:32.822396 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:32.826264 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:32.826270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:32.826277 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:32.826837 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:32.827281 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:32.827285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:32.827289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:32.827486 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:32.828185 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:32.828189 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:32.828396 (+ 207us) client_negotiation.cc:769] Sending connection context
0809 04:50:32.829751 (+ 1355us) client_negotiation.cc:240] Negotiation successful
0809 04:50:32.830223 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":51,"thread_start_us":398,"threads_started":1}
WARNING: no policy specified for oryx/127.31.93.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.93.60@KRBTEST.COM" created.
Aug 09 04:50:32 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251030, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.62@KRBTEST.COM for oryx/127.31.93.61@KRBTEST.COM
I20260809 04:50:32.854055 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:32.837377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53096 (local address 127.31.93.61:41063)
0809 04:50:32.837523 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:32.837528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:32.837847 (+ 319us) server_negotiation.cc:407] Connection header received
0809 04:50:32.838209 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:32.838215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:32.838319 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:32.838390 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:32.839956 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.841033 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.842934 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.843431 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.843524 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:32.847544 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:32.847550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:32.847555 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:32.847592 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:32.850901 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:32.851777 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:32.851782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:32.851786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:32.851957 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:32.852379 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:32.852383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:32.852386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:32.852794 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:32.852929 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:32.853469 (+ 540us) server_negotiation.cc:299] Negotiation successful
0809 04:50:32.853704 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:50:32.854084 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:32.837050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:41063 (local address 127.0.0.1:53096)
0809 04:50:32.837710 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:32.837763 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:32.837984 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:32.838678 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:32.838685 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:32.839509 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:32.839792 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:32.839805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.841180 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.841186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:32.842758 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:32.842778 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.843596 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.843603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:32.843821 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:32.844528 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:32.844552 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:32.847365 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:32.851065 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:32.851080 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:32.851088 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:32.851649 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:32.852083 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:32.852088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:32.852092 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:32.852269 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:32.852924 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:32.852932 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:32.853126 (+ 194us) client_negotiation.cc:769] Sending connection context
0809 04:50:32.853342 (+ 216us) client_negotiation.cc:240] Negotiation successful
0809 04:50:32.853678 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":245,"threads_started":1}
I20260809 04:50:32.856639 9724 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.93.61:41063: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "025ce77beaad4885a83c296e1694006e"
member_type: NON_VOTER
last_known_addr {
host: "127.31.93.61"
port: 41063
}
attrs {
promote: true
}
}
dest_uuid: "391a359744cb4913b6a7a3fdfaff318a"
cas_config_opid_index: -1
I20260809 04:50:32.858263 9724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: NON_VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: true } }
I20260809 04:50:32.866601 10521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:32.866741 10622 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 025ce77beaad4885a83c296e1694006e (127.31.93.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: NON_VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: true } } }
I20260809 04:50:32.871122 10521 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 FOLLOWER]: Refusing update from remote peer 391a359744cb4913b6a7a3fdfaff318a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260809 04:50:32.871645 10622 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.93.61:41063
I20260809 04:50:32.872539 10623 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [LEADER]: Connected to new peer: Peer: permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: NON_VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:32.873245 10621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: NON_VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: true } } }
I20260809 04:50:32.873566 10621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:32.874408 10460 master.cc:537] Master@127.31.93.61:41063 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.93.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.60.keytab.
Entry for principal oryx/127.31.93.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.60.keytab.
I20260809 04:50:32.877388 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.60:35871
--webserver_interface=127.31.93.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.60.keytab
--principal=oryx/127.31.93.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:45495,127.31.93.61:41063,127.31.93.60:35871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:32.878314 9705 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260809 04:50:32.888597 10460 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:32.889127 10460 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:32.889321 10460 tablet_replica.cc:330] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: stopping tablet replica
I20260809 04:50:32.896296 10460 master.cc:552] Master@127.31.93.61:41063 shutdown complete.
I20260809 04:50:32.896428 10460 master_runner.cc:305] Clearing existing system tablet
I20260809 04:50:32.896910 10460 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:32.897056 10460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:32.901151 10460 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:50:32.901211 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:32.900742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:41063 (local address 127.0.0.1:53102)
0809 04:50:32.900936 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:32.901039 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:41063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 04:50:32.901441 9705 consensus_peers.cc:487] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a -> Peer 025ce77beaad4885a83c296e1694006e (127.31.93.61:41063): Couldn't send request to peer 025ce77beaad4885a83c296e1694006e. Status: Network error: Client connection negotiation failed: client connection to 127.31.93.61:41063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 04:50:32.902891 10630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:32.903234 10460 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:32.903640 10460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:50:32.903780 10460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/wal
uuid: "025ce77beaad4885a83c296e1694006e"
format_stamp: "Formatted at 2026-08-09 04:50:32 on dist-test-slave-4tdj"
server_key: "5280d6468025f890c380d8fbe321d806"
server_key_iv: "cce6c544b127496d32c273cc0fa99f16"
server_key_version: "encryptionkey@0"
I20260809 04:50:32.909368 10460 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:32.916191 10460 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 04:50:32.916358 10460 log.cc:1196] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260809 04:50:32.916778 10460 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: Deleting consensus metadata
I20260809 04:50:32.917032 10460 master_runner.cc:315] Copying system tablet from 127.31.93.62:45495
I20260809 04:50:32.919890 10460 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: tablet copy: Beginning tablet copy session from remote peer at address 127.31.93.62:45495
I20260809 04:50:32.935459 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:32.920954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42604 (local address 127.31.93.62:45495)
0809 04:50:32.921123 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:32.921129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:32.921614 (+ 485us) server_negotiation.cc:407] Connection header received
0809 04:50:32.921804 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:32.921810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:32.921908 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:32.922016 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:32.923088 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.924208 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.926003 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.926691 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.926832 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:32.928987 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:32.928994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:32.928999 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:32.929040 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:32.932497 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:32.933270 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:32.933276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:32.933279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:32.933421 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:32.933860 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:32.933865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:32.933869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:32.934238 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:32.934404 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:32.934875 (+ 471us) server_negotiation.cc:299] Negotiation successful
0809 04:50:32.935107 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:50:32.935523 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:32.920931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.0.0.1:42604)
0809 04:50:32.921494 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:32.921513 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:32.921675 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:32.922054 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:32.922058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:32.922630 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:32.922916 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:32.922929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.924375 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.924380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:32.925829 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:32.925845 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:32.926862 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:32.926869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:32.927033 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:32.927719 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:32.927742 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:32.928753 (+ 1011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:32.932642 (+ 3889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:32.932647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:32.932660 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:32.933147 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:32.933539 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:32.933544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:32.933548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:32.933739 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:32.934408 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:32.934413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:32.934598 (+ 185us) client_negotiation.cc:769] Sending connection context
0809 04:50:32.934779 (+ 181us) client_negotiation.cc:240] Negotiation successful
0809 04:50:32.935081 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":323,"threads_started":1}
I20260809 04:50:32.936841 9750 tablet_copy_service.cc:140] P 391a359744cb4913b6a7a3fdfaff318a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 025ce77beaad4885a83c296e1694006e ({username='oryx', principal='oryx/127.31.93.61@KRBTEST.COM'} at 127.0.0.1:42604)
I20260809 04:50:32.937291 9750 tablet_copy_service.cc:161] P 391a359744cb4913b6a7a3fdfaff318a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 025ce77beaad4885a83c296e1694006e at {username='oryx', principal='oryx/127.31.93.61@KRBTEST.COM'} at 127.0.0.1:42604: session id = 025ce77beaad4885a83c296e1694006e-00000000000000000000000000000000
I20260809 04:50:32.942225 9750 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 04:50:32.945111 10460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 04:50:32.952199 10460 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: tablet copy: Starting download of 0 data blocks...
I20260809 04:50:32.952620 10460 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: tablet copy: Starting download of 1 WAL segments...
I20260809 04:50:32.956552 10460 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 04:50:32.961090 9750 tablet_copy_service.cc:342] P 391a359744cb4913b6a7a3fdfaff318a: Request end of tablet copy session 025ce77beaad4885a83c296e1694006e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.93.61@KRBTEST.COM'} at 127.0.0.1:42604
I20260809 04:50:32.961370 9750 tablet_copy_service.cc:433] P 391a359744cb4913b6a7a3fdfaff318a: ending tablet copy session 025ce77beaad4885a83c296e1694006e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 025ce77beaad4885a83c296e1694006e
I20260809 04:50:32.974031 10460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:32.976692 10639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:32.979499 10640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:32.990097 10642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:32.990371 10460 server_base.cc:1034] running on GCE node
I20260809 04:50:32.990804 10460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:32.991472 10460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:32.992703 10460 hybrid_clock.cc:648] HybridClock initialized: now 1786251032992621 us; error 58 us; skew 500 ppm
I20260809 04:50:32.992928 10460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:32 dist-test-slave-4tdj krb5kdc[9655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251032, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:32.995165 10460 init.cc:377] Logged in from keytab as oryx/127.31.93.61@KRBTEST.COM (short username oryx)
I20260809 04:50:32.996403 10460 webserver.cc:460] Webserver started at http://127.31.93.61:46337/ using document root <none> and password file <none>
I20260809 04:50:32.996837 10460 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:32.996932 10460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:33.000126 10460 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:50:33.002152 10652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:33.002626 10460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 04:50:33.002843 10460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/wal
uuid: "025ce77beaad4885a83c296e1694006e"
format_stamp: "Formatted at 2026-08-09 04:50:32 on dist-test-slave-4tdj"
server_key: "5280d6468025f890c380d8fbe321d806"
server_key_iv: "cce6c544b127496d32c273cc0fa99f16"
server_key_version: "encryptionkey@0"
I20260809 04:50:33.003077 10460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:33.014114 10460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:33.018134 10460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:33.082728 10460 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.61:41063
I20260809 04:50:33.082748 10751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.61:41063 every 8 connection(s)
I20260809 04:50:33.083705 10460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-1/data/info.pb
I20260809 04:50:33.089468 10754 sys_catalog.cc:263] Verifying existing consensus state
I20260809 04:50:33.093689 10754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: Bootstrap starting.
I20260809 04:50:33.118924 10754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:50:33.119652 10754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: Bootstrap complete.
I20260809 04:50:33.120975 10754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: NON_VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: true } }
I20260809 04:50:33.121201 10754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 025ce77beaad4885a83c296e1694006e, State: Initialized, Role: LEARNER
I20260809 04:50:33.121548 10754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: NON_VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: true } }
I20260809 04:50:33.122545 10757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: NON_VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: true } } }
I20260809 04:50:33.123011 10754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:33.123072 10757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: This master's current role is: LEARNER
I20260809 04:50:33.135617 10770 catalog_manager.cc:1259] Loaded cluster ID: bbc24b55091f4398b1e1f66b7eb68697
I20260809 04:50:33.135704 10770 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: loading cluster ID for follower catalog manager: success
I20260809 04:50:33.138422 10770 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: acquiring CA information for follower catalog manager: success
I20260809 04:50:33.140682 10770 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 04:50:33.275892 10454 heartbeater.cc:503] Master 127.31.93.62:45495 was elected leader, sending a full tablet report...
I20260809 04:50:33.325742 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.317984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:41063 (local address 127.0.0.1:53106)
0809 04:50:33.318262 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:33.318284 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:33.318493 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:33.319836 (+ 1343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:33.319841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:33.319870 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:33.320099 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.320115 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.321676 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.321680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.323077 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.323092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.324852 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.324858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.325055 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.325121 (+ 66us) client_negotiation.cc:769] Sending connection context
0809 04:50:33.325218 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 04:50:33.325381 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260809 04:50:33.325827 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.318347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53106 (local address 127.31.93.61:41063)
0809 04:50:33.319437 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:33.319443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:33.319500 (+ 57us) server_negotiation.cc:407] Connection header received
0809 04:50:33.319567 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:33.319571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:33.319639 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:33.319840 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:33.320261 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.321492 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.323217 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.324699 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.324812 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.325081 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:33.325263 (+ 182us) server_negotiation.cc:299] Negotiation successful
0809 04:50:33.325394 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":479,"threads_started":1}
I20260809 04:50:33.327387 10710 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260809 04:50:33.328306 10757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 391a359744cb4913b6a7a3fdfaff318a. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: NON_VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: true } } }
I20260809 04:50:33.328613 10757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: This master's current role is: LEARNER
W20260809 04:50:33.428416 10627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:33.428725 10627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:33.428881 10627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:33.437654 10627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:33.437778 10627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:33.437881 10627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:33.438050 10627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:33.438172 10627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:50:33.450142 10627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:45495,127.31.93.61:41063,127.31.93.60:35871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.60.keytab
--principal=oryx/127.31.93.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.60:35871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.93.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:33.451669 10627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:33.453256 10627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:33.468053 10780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:33.467967 10782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:33.468217 10788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:33.469731 10627 server_base.cc:1034] running on GCE node
I20260809 04:50:33.470539 10627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:33.471925 10627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:33.473127 10627 hybrid_clock.cc:648] HybridClock initialized: now 1786251033473074 us; error 43 us; skew 500 ppm
I20260809 04:50:33.473455 10627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:33 dist-test-slave-4tdj krb5kdc[9655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251033, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:33.481071 10627 init.cc:377] Logged in from keytab as oryx/127.31.93.60@KRBTEST.COM (short username oryx)
I20260809 04:50:33.482896 10627 webserver.cc:460] Webserver started at http://127.31.93.60:39195/ using document root <none> and password file <none>
I20260809 04:50:33.483579 10627 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:33.483788 10627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:33.484172 10627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:33.487689 10627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/data/instance:
uuid: "e0326546ecbb435d8d754a858e366376"
format_stamp: "Formatted at 2026-08-09 04:50:33 on dist-test-slave-4tdj"
server_key: "e67a0127b0ea5db3f78ec5b4cbe2e294"
server_key_iv: "d31b4f6b34b7146c05ba3cc5874c3557"
server_key_version: "encryptionkey@0"
I20260809 04:50:33.488454 10627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/wal/instance:
uuid: "e0326546ecbb435d8d754a858e366376"
format_stamp: "Formatted at 2026-08-09 04:50:33 on dist-test-slave-4tdj"
server_key: "e67a0127b0ea5db3f78ec5b4cbe2e294"
server_key_iv: "d31b4f6b34b7146c05ba3cc5874c3557"
server_key_version: "encryptionkey@0"
I20260809 04:50:33.494407 10627 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260809 04:50:33.498310 10799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:33.499608 10627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:50:33.499827 10627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/wal
uuid: "e0326546ecbb435d8d754a858e366376"
format_stamp: "Formatted at 2026-08-09 04:50:33 on dist-test-slave-4tdj"
server_key: "e67a0127b0ea5db3f78ec5b4cbe2e294"
server_key_iv: "d31b4f6b34b7146c05ba3cc5874c3557"
server_key_version: "encryptionkey@0"
I20260809 04:50:33.500069 10627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:33.527774 10627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:33.533979 10627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:33.534535 10627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:33.573390 10627 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.60:35871
I20260809 04:50:33.573415 10887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.60:35871 every 8 connection(s)
I20260809 04:50:33.574841 10627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/data/info.pb
I20260809 04:50:33.579095 10888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:33.583029 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 10627
I20260809 04:50:33.588104 10888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:50:33 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.93.60@KRBTEST.COM
Aug 09 04:50:33 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251033, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.60@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM
I20260809 04:50:33.609236 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.591860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.31.93.62:45495)
0809 04:50:33.593312 (+ 1452us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:33.593317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:33.593332 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:50:33.593379 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:33.593383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:33.593436 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:33.593562 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:33.595161 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.596180 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.598128 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.598812 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.598923 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.603070 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:33.603074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:33.603078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:33.603110 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:33.606189 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.606956 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.606962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.606966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.607168 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.607536 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.607539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.607541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.607868 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:33.608039 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:33.608491 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 04:50:33.608687 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1356,"thread_start_us":1260,"threads_started":1}
I20260809 04:50:33.609429 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.585493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60194 (local address 127.31.93.60:35871)
0809 04:50:33.586546 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:33.586557 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:33.586601 (+ 44us) server_negotiation.cc:407] Connection header received
0809 04:50:33.588976 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:33.588999 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:33.589483 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:33.590047 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:33.592230 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.593964 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.595508 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.596213 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.596621 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.601459 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:33.601466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:33.601477 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:33.601506 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:33.605168 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.605925 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.605933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.605941 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.606094 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.606604 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.606609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.606613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.607090 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:33.607385 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:33.607862 (+ 477us) server_negotiation.cc:299] Negotiation successful
0809 04:50:33.608257 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":858,"thread_start_us":569,"threads_started":1}
I20260809 04:50:33.609503 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.591521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.0.0.1:42620)
0809 04:50:33.592539 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:33.592595 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:33.592896 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:33.593707 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:33.593714 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:33.594665 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:33.595007 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.595017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.596321 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.596325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.597991 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.598004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.599108 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.599116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.599244 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.599827 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:33.599854 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:33.602870 (+ 3016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:33.606341 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:33.606352 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:33.606360 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:33.606842 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:33.607282 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:33.607287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:33.607290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:33.607425 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:33.607998 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:33.608006 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:33.608164 (+ 158us) client_negotiation.cc:769] Sending connection context
0809 04:50:33.608322 (+ 158us) client_negotiation.cc:240] Negotiation successful
0809 04:50:33.608628 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":5,"thread_start_us":610,"threads_started":1}
I20260809 04:50:33.611534 10888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:50:33 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251033, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.60@KRBTEST.COM for oryx/127.31.93.61@KRBTEST.COM
I20260809 04:50:33.624574 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.612355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:41063 (local address 127.0.0.1:53116)
0809 04:50:33.612602 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:33.612616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:33.612711 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:33.612988 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:33.612992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:33.613416 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:33.613547 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.613556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.614749 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.614753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.615641 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.615652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.616519 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.616523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.616733 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.617219 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:33.617235 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:33.619556 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:33.622540 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:33.622543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:33.622546 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:33.622963 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:33.623305 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:33.623309 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:33.623311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:33.623435 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:33.623896 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:33.623899 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:33.624014 (+ 115us) client_negotiation.cc:769] Sending connection context
0809 04:50:33.624120 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 04:50:33.624365 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260809 04:50:33.624675 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.612325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.31.93.61:41063)
0809 04:50:33.612505 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:33.612511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:33.612680 (+ 169us) server_negotiation.cc:407] Connection header received
0809 04:50:33.612810 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:33.612813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:33.612877 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:33.612956 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:33.613702 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.614637 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.615753 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.616369 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.616461 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.619721 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:33.619726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:33.619730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:33.619762 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:33.622385 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.623049 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.623053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.623055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.623207 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.623510 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.623513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.623515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.623804 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:33.623914 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:33.624188 (+ 274us) server_negotiation.cc:299] Negotiation successful
0809 04:50:33.624394 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 04:50:33.625888 10888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:50:33 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251033, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.60@KRBTEST.COM for oryx/127.31.93.60@KRBTEST.COM
I20260809 04:50:33.638985 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.626718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:35871 (local address 127.0.0.1:60202)
0809 04:50:33.626864 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:33.626881 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:33.626980 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:33.627266 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:33.627270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:33.627807 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:33.627963 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.627971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.629129 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.629134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.630065 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.630077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.630831 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.630836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.631020 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.631480 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:33.631495 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:33.634094 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:33.636642 (+ 2548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:33.636646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:33.636648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:33.637120 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:33.637476 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:33.637479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:33.637481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:33.637675 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:33.638146 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:33.638149 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:33.638258 (+ 109us) client_negotiation.cc:769] Sending connection context
0809 04:50:33.638480 (+ 222us) client_negotiation.cc:240] Negotiation successful
0809 04:50:33.638671 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 04:50:33.639045 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.626701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60202 (local address 127.31.93.60:35871)
0809 04:50:33.626865 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:33.626870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:33.626955 (+ 85us) server_negotiation.cc:407] Connection header received
0809 04:50:33.627077 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:33.627082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:33.627137 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:33.627215 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:33.628084 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.629019 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.630196 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.630698 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.630810 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.634266 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:33.634271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:33.634274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:33.634337 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:33.636521 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.637217 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.637221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.637223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.637395 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.637768 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.637772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.637773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.638052 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:33.638192 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:33.638569 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 04:50:33.638770 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":52}
I20260809 04:50:33.644335 10888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: Bootstrap starting.
I20260809 04:50:33.648265 10888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:33.649616 10888 log.cc:824] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:33.652794 10888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: No bootstrap required, opened a new log
I20260809 04:50:33.663076 10888 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } }
I20260809 04:50:33.663516 10888 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:33.663671 10888 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0326546ecbb435d8d754a858e366376, State: Initialized, Role: FOLLOWER
I20260809 04:50:33.664235 10888 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } }
I20260809 04:50:33.665272 10896 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } } }
I20260809 04:50:33.665747 10896 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:50:33.666344 10888 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 04:50:33.678371 10914 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:50:33.678596 10914 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:50:33.682235 10627 master_runner.cc:428] Detected that this master e0326546ecbb435d8d754a858e366376 is joining an existing cluster
I20260809 04:50:33.682370 10627 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.93.60:35871
I20260809 04:50:33.711290 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.688350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53130 (local address 127.31.93.61:41063)
0809 04:50:33.688553 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:33.688559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:33.691461 (+ 2902us) server_negotiation.cc:407] Connection header received
0809 04:50:33.691772 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:33.691779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:33.691865 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:33.691934 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:33.694928 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.696335 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.698832 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.699392 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.699480 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.702917 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:33.702924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:33.702929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:33.702966 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:33.707712 (+ 4746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.708514 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.708520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.708524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.708708 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.709197 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.709203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.709206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.709595 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:33.709728 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:33.710469 (+ 741us) server_negotiation.cc:299] Negotiation successful
0809 04:50:33.710752 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:50:33.713068 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.688325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:41063 (local address 127.0.0.1:53130)
0809 04:50:33.691202 (+ 2877us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:33.691223 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:33.691613 (+ 390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:33.692227 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:33.692232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:33.694016 (+ 1784us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:33.694742 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.694758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.696589 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.696595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.698651 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.698667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.699860 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.699867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.700053 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.700797 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:33.700828 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:33.702669 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:33.707878 (+ 5209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:33.707884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:33.707888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:33.708381 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:33.708850 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:33.708855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:33.708859 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:33.709072 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:33.709823 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:33.709827 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:33.710131 (+ 304us) client_negotiation.cc:769] Sending connection context
0809 04:50:33.711563 (+ 1432us) client_negotiation.cc:240] Negotiation successful
0809 04:50:33.711948 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2710,"thread_start_us":16529,"threads_started":1}
I20260809 04:50:33.715759 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 9814
I20260809 04:50:33.722216 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.690370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.31.93.60:35871)
0809 04:50:33.690514 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:33.690519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:33.699331 (+ 8812us) server_negotiation.cc:407] Connection header received
0809 04:50:33.699483 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:33.699488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:33.699561 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:33.699665 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:33.701686 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.703072 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.705496 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.706280 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.706397 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.711071 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:33.711086 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:33.711091 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:33.711123 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:33.714691 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.719752 (+ 5061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.719758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.719762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.719991 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.720660 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.720665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.720669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.721008 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:33.721137 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:33.721575 (+ 438us) server_negotiation.cc:299] Negotiation successful
0809 04:50:33.721799 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 04:50:33.723232 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.688954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:35871 (local address 127.0.0.1:60218)
0809 04:50:33.699055 (+ 10101us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:33.699076 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:33.699370 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:33.699848 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:33.699852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:33.701092 (+ 1240us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:33.701523 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.701540 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.703280 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.703286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.705348 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.705363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.706843 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.706850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.707041 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.709262 (+ 2221us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:33.709289 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:33.710902 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:33.719012 (+ 8110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:33.719018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:33.719022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:33.719625 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:33.720395 (+ 770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:33.720400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:33.720404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:33.720557 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:33.721209 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:33.721214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:33.721356 (+ 142us) client_negotiation.cc:769] Sending connection context
0809 04:50:33.722412 (+ 1056us) client_negotiation.cc:240] Negotiation successful
0809 04:50:33.722720 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9938,"thread_start_us":9685,"threads_started":1}
I20260809 04:50:33.732859 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.687937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.31.93.62:45495)
0809 04:50:33.688111 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:33.688117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:33.705505 (+ 17388us) server_negotiation.cc:407] Connection header received
0809 04:50:33.705758 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:33.705764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:33.705848 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:33.705924 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:33.707640 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.708986 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.711185 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.712062 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.712151 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.717951 (+ 5800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:33.717957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:33.717962 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:33.718000 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:33.721814 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.729837 (+ 8023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.729844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.729848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.730036 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.730603 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.730609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.730613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.731079 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:33.731220 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:33.731997 (+ 777us) server_negotiation.cc:299] Negotiation successful
0809 04:50:33.732288 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:50:33.733911 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.689220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.0.0.1:42634)
0809 04:50:33.705268 (+ 16048us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:33.705285 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:33.705613 (+ 328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:33.706150 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:33.706156 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:33.706978 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:33.707495 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.707509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.709142 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.709147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.711026 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.711043 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.712659 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.712666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.712850 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.713548 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:33.713569 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:33.717755 (+ 4186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:33.728489 (+ 10734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:33.728495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:33.728499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:33.729013 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:33.730179 (+ 1166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:33.730185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:33.730189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:33.730448 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:33.731308 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:33.731313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:33.731697 (+ 384us) client_negotiation.cc:769] Sending connection context
0809 04:50:33.733038 (+ 1341us) client_negotiation.cc:240] Negotiation successful
0809 04:50:33.733349 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15577,"thread_start_us":3566,"threads_started":1}
Aug 09 04:50:33 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.93.1@KRBTEST.COM
Aug 09 04:50:33 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251030, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.62@KRBTEST.COM for oryx/127.31.93.60@KRBTEST.COM
I20260809 04:50:33.749733 10930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: NON_VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: true } } }
I20260809 04:50:33.751495 10930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:33.753830 10621 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a: attempting to promote NON_VOTER 025ce77beaad4885a83c296e1694006e to VOTER
I20260809 04:50:33.755336 10621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } }
I20260809 04:50:33.757854 10710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 LEARNER]: Refusing update from remote peer 391a359744cb4913b6a7a3fdfaff318a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 04:50:33.758270 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.717781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.31.93.1:33309)
0809 04:50:33.732016 (+ 14235us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:33.732028 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:33.732060 (+ 32us) server_negotiation.cc:407] Connection header received
0809 04:50:33.732132 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:33.732142 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:33.732470 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:33.735266 (+ 2796us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:33.735667 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.737504 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.739514 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.740690 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.740837 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.747753 (+ 6916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:33.747771 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:33.747793 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:33.747877 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:33.753659 (+ 5782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.754780 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.754789 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.754792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.755072 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.755613 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.755617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.755620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.756062 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:33.757305 (+ 1243us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:33.757470 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 04:50:33.757770 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13975,"thread_start_us":14982,"threads_started":1}
I20260809 04:50:33.759066 10621 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [LEADER]: Connected to new peer: Peer: permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 04:50:33.759954 10005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:33.762038 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.740441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.31.93.60:35871)
0809 04:50:33.740581 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:33.740587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:33.740604 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:50:33.740689 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:33.740693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:33.740787 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:33.740872 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:33.741982 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.743095 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.745265 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.746222 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.746392 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.751257 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:33.751264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:33.751269 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:33.751305 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:33.755683 (+ 4378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.756638 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.756643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.756646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.756788 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.757394 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.757400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.757403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.757767 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:33.757917 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:33.761463 (+ 3546us) server_negotiation.cc:299] Negotiation successful
0809 04:50:33.761684 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 04:50:33.763183 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.739925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:35871 (local address 127.0.0.1:60226)
0809 04:50:33.740167 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:33.740186 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:33.740288 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:33.740890 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:33.740895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:33.741519 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:33.741807 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.741823 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.743265 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.743271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.745014 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.745032 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.746686 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.746694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.746914 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.747564 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:33.747588 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:33.751040 (+ 3452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:33.755854 (+ 4814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:33.755859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:33.755862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:33.756484 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:33.756924 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:33.756928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:33.756932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:33.757257 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:33.758014 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:33.758020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:33.758209 (+ 189us) client_negotiation.cc:769] Sending connection context
0809 04:50:33.762398 (+ 4189us) client_negotiation.cc:240] Negotiation successful
0809 04:50:33.762737 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260809 04:50:33.764303 10710 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 025ce77beaad4885a83c296e1694006e (127.31.93.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } }
I20260809 04:50:33.768947 10757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } }
I20260809 04:50:33.769129 10757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:50:33.770207 9724 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.93.60:35871: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e0326546ecbb435d8d754a858e366376"
member_type: NON_VOTER
last_known_addr {
host: "127.31.93.60"
port: 35871
}
attrs {
promote: true
}
}
dest_uuid: "391a359744cb4913b6a7a3fdfaff318a"
cas_config_opid_index: 5
W20260809 04:50:33.770888 9724 master_service.cc:301] AddMaster 127.31.93.60:35871 failed with retriable error: Illegal state: Failed initiating master Raft ChangeConfig request, error: unknown: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: NON_VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } }
W20260809 04:50:33.772522 10917 master_proxy_rpc.cc:203] Re-attempting AddMaster request to leader Master (127.31.93.62:45495)
I20260809 04:50:33.774238 10933 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 025ce77beaad4885a83c296e1694006e (127.31.93.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } }
I20260809 04:50:33.777928 10930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } }
I20260809 04:50:33.778261 10930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:33.778851 10937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } }
I20260809 04:50:33.779260 10937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:33.779747 9724 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.93.60:35871: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e0326546ecbb435d8d754a858e366376"
member_type: NON_VOTER
last_known_addr {
host: "127.31.93.60"
port: 35871
}
attrs {
promote: true
}
}
dest_uuid: "391a359744cb4913b6a7a3fdfaff318a"
cas_config_opid_index: 6
I20260809 04:50:33.780432 9724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: NON_VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: true } }
I20260809 04:50:33.782744 10710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 FOLLOWER]: Refusing update from remote peer 391a359744cb4913b6a7a3fdfaff318a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 04:50:33.783070 10844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:33.783529 10937 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [LEADER]: Connected to new peer: Peer: permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 04:50:33.786319 10930 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e0326546ecbb435d8d754a858e366376 (127.31.93.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: NON_VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: true } } }
I20260809 04:50:33.787638 10844 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 1 FOLLOWER]: Refusing update from remote peer 391a359744cb4913b6a7a3fdfaff318a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 04:50:33.788482 10621 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: NON_VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 04:50:33.790719 10808 tablet.cc:2369] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:50:33.793275 10930 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.93.60:35871
I20260809 04:50:33.793715 10627 master.cc:537] Master@127.31.93.60:35871 shutting down...
I20260809 04:50:33.793838 10621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: NON_VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: true } } }
I20260809 04:50:33.794119 10621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:33.794742 10621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: NON_VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: true } } }
I20260809 04:50:33.795079 10621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:33.795948 10710 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e0326546ecbb435d8d754a858e366376 (127.31.93.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: NON_VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: true } } }
I20260809 04:50:33.800196 10757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: NON_VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: true } } }
I20260809 04:50:33.800640 10757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:50:33.807133 10841 raft_consensus.cc:858] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: NON_VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } }
W20260809 04:50:33.807255 10841 raft_consensus.cc:1565] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: NON_VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } }
I20260809 04:50:33.834630 10841 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 025ce77beaad4885a83c296e1694006e (127.31.93.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: NON_VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: true } } }
I20260809 04:50:33.838802 10947 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: NON_VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: true } } }
I20260809 04:50:33.839260 10947 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 04:50:33.839805 10896 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 391a359744cb4913b6a7a3fdfaff318a. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: NON_VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: true } } }
I20260809 04:50:33.840016 10896 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 04:50:33.840134 10944 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: NON_VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: true } } }
I20260809 04:50:33.840332 10944 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 04:50:33.841375 10933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: NON_VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: true } } }
I20260809 04:50:33.841670 10933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:33.842538 10627 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260809 04:50:33.842973 10627 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260809 04:50:33.843160 10627 tablet_replica.cc:330] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: stopping tablet replica
W20260809 04:50:33.843863 9704 consensus_peers.cc:487] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a -> Peer e0326546ecbb435d8d754a858e366376 (127.31.93.60:35871): Couldn't send request to peer e0326546ecbb435d8d754a858e366376. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 04:50:33.851225 10627 master.cc:552] Master@127.31.93.60:35871 shutdown complete.
I20260809 04:50:33.851312 10627 master_runner.cc:305] Clearing existing system tablet
I20260809 04:50:33.851676 10627 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:33.851790 10627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:33.856097 10627 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 04:50:33.858641 10952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:33.858949 10627 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:33.859277 10627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:50:33.859418 10627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/wal
uuid: "e0326546ecbb435d8d754a858e366376"
format_stamp: "Formatted at 2026-08-09 04:50:33 on dist-test-slave-4tdj"
server_key: "e67a0127b0ea5db3f78ec5b4cbe2e294"
server_key_iv: "d31b4f6b34b7146c05ba3cc5874c3557"
server_key_version: "encryptionkey@0"
I20260809 04:50:33.864032 10627 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:33.870849 10627 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 04:50:33.871050 10627 log.cc:1196] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260809 04:50:33.871615 10627 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: Deleting consensus metadata
I20260809 04:50:33.871941 10627 master_runner.cc:315] Copying system tablet from 127.31.93.62:45495
I20260809 04:50:33.884402 10627 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: tablet copy: Beginning tablet copy session from remote peer at address 127.31.93.62:45495
I20260809 04:50:33.903793 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.885325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.0.0.1:42636)
0809 04:50:33.886373 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:33.886395 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:33.886611 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:33.887172 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:33.887177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:33.887776 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:33.888083 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.888098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.889588 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.889593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.891326 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:33.891343 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.892591 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.892597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:33.892784 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.893680 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:33.893704 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:33.895201 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:33.899658 (+ 4457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:33.899664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:33.899668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:33.900163 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:33.900663 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:33.900667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:33.900671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:33.901670 (+ 999us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:33.902419 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:33.902424 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:33.902694 (+ 270us) client_negotiation.cc:769] Sending connection context
0809 04:50:33.902847 (+ 153us) client_negotiation.cc:240] Negotiation successful
0809 04:50:33.903216 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"thread_start_us":736,"threads_started":1}
I20260809 04:50:33.904280 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:33.885385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42636 (local address 127.31.93.62:45495)
0809 04:50:33.885582 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:33.885589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:33.886492 (+ 903us) server_negotiation.cc:407] Connection header received
0809 04:50:33.886785 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:33.886791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:33.886878 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:33.886950 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:33.888268 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.889390 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.891508 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:33.892262 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:33.892355 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:33.895403 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:33.895410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:33.895416 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:33.895458 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:33.899398 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.900354 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.900360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.900365 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.900521 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:33.901805 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:33.901811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:33.901814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:33.902197 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:33.902355 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:33.903074 (+ 719us) server_negotiation.cc:299] Negotiation successful
0809 04:50:33.903362 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 04:50:33.904964 9750 tablet_copy_service.cc:140] P 391a359744cb4913b6a7a3fdfaff318a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e0326546ecbb435d8d754a858e366376 ({username='oryx', principal='oryx/127.31.93.60@KRBTEST.COM'} at 127.0.0.1:42636)
I20260809 04:50:33.905153 9750 tablet_copy_service.cc:161] P 391a359744cb4913b6a7a3fdfaff318a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e0326546ecbb435d8d754a858e366376 at {username='oryx', principal='oryx/127.31.93.60@KRBTEST.COM'} at 127.0.0.1:42636: session id = e0326546ecbb435d8d754a858e366376-00000000000000000000000000000000
I20260809 04:50:33.907830 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 9814
I20260809 04:50:33.911477 9750 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 04:50:33.913447 10627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 04:50:33.920336 10627 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: tablet copy: Starting download of 0 data blocks...
I20260809 04:50:33.920629 10627 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: tablet copy: Starting download of 1 WAL segments...
I20260809 04:50:33.923019 10627 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 04:50:33.927204 9750 tablet_copy_service.cc:342] P 391a359744cb4913b6a7a3fdfaff318a: Request end of tablet copy session e0326546ecbb435d8d754a858e366376-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.93.60@KRBTEST.COM'} at 127.0.0.1:42636
I20260809 04:50:33.927341 9750 tablet_copy_service.cc:433] P 391a359744cb4913b6a7a3fdfaff318a: ending tablet copy session e0326546ecbb435d8d754a858e366376-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e0326546ecbb435d8d754a858e366376
I20260809 04:50:33.932183 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:33309
--local_ip_for_outbound_sockets=127.31.93.1
--tserver_master_addrs=127.31.93.62:45495
--webserver_port=46327
--webserver_interface=127.31.93.1
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.1.keytab
--principal=oryx/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.93.62:45495,127.31.93.61:41063
--tserver_master_addrs=127.31.93.62:45495,127.31.93.61:41063,127.31.93.60:35871 with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:50:33.938400 10627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:33.940860 10962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:33.941876 10963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:33.949172 10627 server_base.cc:1034] running on GCE node
W20260809 04:50:33.949726 10966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:33.950479 10627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:33.951130 10627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:33.952337 10627 hybrid_clock.cc:648] HybridClock initialized: now 1786251033952307 us; error 66 us; skew 500 ppm
I20260809 04:50:33.952577 10627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:33 dist-test-slave-4tdj krb5kdc[9655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251033, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:33.954773 10627 init.cc:377] Logged in from keytab as oryx/127.31.93.60@KRBTEST.COM (short username oryx)
I20260809 04:50:33.962252 10627 webserver.cc:460] Webserver started at http://127.31.93.60:44893/ using document root <none> and password file <none>
I20260809 04:50:33.962601 10627 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:33.962706 10627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:33.965675 10627 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:50:33.967628 10977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:33.968071 10627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 04:50:33.968240 10627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/wal
uuid: "e0326546ecbb435d8d754a858e366376"
format_stamp: "Formatted at 2026-08-09 04:50:33 on dist-test-slave-4tdj"
server_key: "e67a0127b0ea5db3f78ec5b4cbe2e294"
server_key_iv: "d31b4f6b34b7146c05ba3cc5874c3557"
server_key_version: "encryptionkey@0"
I20260809 04:50:33.968472 10627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:33.992800 10627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:33.997071 10627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:34.020509 10627 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.60:35871
I20260809 04:50:34.020627 11053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.60:35871 every 8 connection(s)
I20260809 04:50:34.021828 10627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/master-2/data/info.pb
I20260809 04:50:34.027493 11054 sys_catalog.cc:263] Verifying existing consensus state
I20260809 04:50:34.031746 11054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: Bootstrap starting.
I20260809 04:50:34.046838 11054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:50:34.047366 11054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: Bootstrap complete.
I20260809 04:50:34.048452 11054 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: NON_VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: true } }
I20260809 04:50:34.048754 11054 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e0326546ecbb435d8d754a858e366376, State: Initialized, Role: LEARNER
I20260809 04:50:34.049003 11054 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: NON_VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: true } }
I20260809 04:50:34.050414 11060 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: NON_VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: true } } }
I20260809 04:50:34.050727 11054 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:34.050895 11060 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: This master's current role is: LEARNER
I20260809 04:50:34.058804 11076 catalog_manager.cc:1259] Loaded cluster ID: bbc24b55091f4398b1e1f66b7eb68697
I20260809 04:50:34.058986 11076 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: loading cluster ID for follower catalog manager: success
I20260809 04:50:34.062577 11076 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: acquiring CA information for follower catalog manager: success
I20260809 04:50:34.064927 11076 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 04:50:34.332556 10961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:34.332819 10961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:34.333060 10961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:34.341390 10961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:34.341508 10961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:34.341663 10961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:50:34.353497 10961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.1.keytab
--principal=oryx/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:33309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=46327
--tserver_master_addrs=127.31.93.62:45495,127.31.93.61:41063,127.31.93.60:35871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:34.354859 10961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:34.356153 10961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:34.370610 11083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:34.370407 11088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:34.370687 11084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:34.371896 10961 server_base.cc:1034] running on GCE node
I20260809 04:50:34.372642 10961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:34.373804 10961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:34.375056 10961 hybrid_clock.cc:648] HybridClock initialized: now 1786251034374969 us; error 29 us; skew 500 ppm
I20260809 04:50:34.375478 10961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:34 dist-test-slave-4tdj krb5kdc[9655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251034, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:34.379545 10961 init.cc:377] Logged in from keytab as oryx/127.31.93.1@KRBTEST.COM (short username oryx)
I20260809 04:50:34.380856 10961 webserver.cc:460] Webserver started at http://127.31.93.1:46327/ using document root <none> and password file <none>
I20260809 04:50:34.381448 10961 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:34.381572 10961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:34.388418 10961 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 04:50:34.392030 11099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:34.393688 10961 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:50:34.393939 10961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "67bceba2f21444d69fdc7d07a7242c3c"
format_stamp: "Formatted at 2026-08-09 04:50:31 on dist-test-slave-4tdj"
server_key: "35d36a65af47c9313a7f668571cf150e"
server_key_iv: "e81664379b2c222d7a107d6b95c4e69f"
server_key_version: "encryptionkey@0"
I20260809 04:50:34.395289 10961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:34.419143 10961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:34.423782 10961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:34.424100 10961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:34.425417 10961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:34.426935 10961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:34.427160 10961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:34.427328 10961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:34.427507 10961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:34.463583 10937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: NON_VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: true } } }
I20260809 04:50:34.464450 10937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:34.474870 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:34.463744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:35871 (local address 127.0.0.1:60232)
0809 04:50:34.465596 (+ 1852us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:34.465622 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:34.465781 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:34.467216 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:34.467223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:34.467256 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:34.467454 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:34.467468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.469433 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.469439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:34.471484 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:34.471500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.473932 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.473940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:34.474258 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:34.474322 (+ 64us) client_negotiation.cc:769] Sending connection context
0809 04:50:34.474418 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 04:50:34.474577 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1719,"thread_start_us":1842,"threads_started":1}
I20260809 04:50:34.475446 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:34.464135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60232 (local address 127.31.93.60:35871)
0809 04:50:34.466801 (+ 2666us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:34.466807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:34.466830 (+ 23us) server_negotiation.cc:407] Connection header received
0809 04:50:34.466913 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:34.466919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:34.467028 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:34.467598 (+ 570us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:34.467971 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.469270 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.471819 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.473615 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.473800 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:34.474094 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:34.475091 (+ 997us) server_negotiation.cc:299] Negotiation successful
0809 04:50:34.475202 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2503,"thread_start_us":2347,"threads_started":1}
I20260809 04:50:34.476410 11022 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260809 04:50:34.476953 11060 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 391a359744cb4913b6a7a3fdfaff318a. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: NON_VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: true } } }
I20260809 04:50:34.477202 11060 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: This master's current role is: LEARNER
I20260809 04:50:34.477837 11214 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a: attempting to promote NON_VOTER e0326546ecbb435d8d754a858e366376 to VOTER
I20260809 04:50:34.478674 11214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } }
I20260809 04:50:34.480958 11022 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 1 LEARNER]: Refusing update from remote peer 391a359744cb4913b6a7a3fdfaff318a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 04:50:34.480955 10710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 FOLLOWER]: Refusing update from remote peer 391a359744cb4913b6a7a3fdfaff318a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 04:50:34.481758 11214 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 04:50:34.482244 11213 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [LEADER]: Connected to new peer: Peer: permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 04:50:34.486027 10937 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e0326546ecbb435d8d754a858e366376 (127.31.93.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } } }
I20260809 04:50:34.487568 10710 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e0326546ecbb435d8d754a858e366376 (127.31.93.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } } }
I20260809 04:50:34.490964 11213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } } }
I20260809 04:50:34.491374 11213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:34.491775 11213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } } }
I20260809 04:50:34.492070 11213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:34.492193 10961 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:33309
I20260809 04:50:34.492427 11213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } } }
I20260809 04:50:34.492647 11213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:34.492007 11022 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e0326546ecbb435d8d754a858e366376 (127.31.93.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } } }
I20260809 04:50:34.492614 11277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:33309 every 8 connection(s)
I20260809 04:50:34.493917 10961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:50:34.496871 11060 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } } }
I20260809 04:50:34.497143 11060 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:50:34.500633 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 10961
I20260809 04:50:34.501102 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 10023
I20260809 04:50:34.519570 11281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "391a359744cb4913b6a7a3fdfaff318a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } } }
I20260809 04:50:34.519960 11281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: This master's current role is: FOLLOWER
Aug 09 04:50:34 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251034, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.60@KRBTEST.COM
Aug 09 04:50:34 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251034, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM
Aug 09 04:50:34 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.93.2@KRBTEST.COM
I20260809 04:50:34.558841 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:34.498621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:57701 (local address 127.31.93.60:35871)
0809 04:50:34.498826 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:34.498833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:34.525486 (+ 26653us) server_negotiation.cc:407] Connection header received
0809 04:50:34.529155 (+ 3669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:34.529162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:34.529255 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:34.529330 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:34.532201 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.533322 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.536440 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.537271 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.537389 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:34.550598 (+ 13209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:34.550606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:34.550611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:34.550655 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:34.554654 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:34.555671 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:34.555677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:34.555680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:34.555878 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:34.556790 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:34.556795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:34.556798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:34.557157 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:34.557287 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:34.558234 (+ 947us) server_negotiation.cc:299] Negotiation successful
0809 04:50:34.558494 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Aug 09 04:50:34 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251034, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.61@KRBTEST.COM
I20260809 04:50:34.561305 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:34.497957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:35871 (local address 127.31.93.1:57701)
0809 04:50:34.525117 (+ 27160us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:34.525170 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:34.528802 (+ 3632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:34.529887 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:34.529899 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:34.530892 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:34.531929 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:34.531962 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.533538 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.533544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:34.536257 (+ 2713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:34.536273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.537785 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.537795 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:34.538207 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:34.539461 (+ 1254us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:34.545996 (+ 6535us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:34.550198 (+ 4202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:34.554856 (+ 4658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:34.554864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:34.554935 (+ 71us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:34.555526 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:34.556069 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:34.556074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:34.556078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:34.556652 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:34.557385 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:34.557393 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:34.557797 (+ 404us) client_negotiation.cc:769] Sending connection context
0809 04:50:34.559287 (+ 1490us) client_negotiation.cc:240] Negotiation successful
0809 04:50:34.559840 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26737,"thread_start_us":26374,"threads_started":1}
I20260809 04:50:34.562541 11290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:34.503284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.31.93.2:40459)
0809 04:50:34.533260 (+ 29976us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:34.533269 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:34.533298 (+ 29us) server_negotiation.cc:407] Connection header received
0809 04:50:34.533378 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:34.533387 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:34.533660 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:34.534013 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:34.538200 (+ 4187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.542684 (+ 4484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.544933 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.545670 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.545786 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:34.554215 (+ 8429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:34.554230 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:34.554247 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:34.554318 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:34.558936 (+ 4618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:34.559962 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:34.559970 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:34.559974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:34.560145 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:34.560604 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:34.560608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:34.560611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:34.560992 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:34.561276 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:34.561829 (+ 553us) server_negotiation.cc:299] Negotiation successful
0809 04:50:34.562102 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29803,"thread_start_us":31839,"threads_started":1}
I20260809 04:50:34.562875 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:34.500426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.1:60995)
0809 04:50:34.535169 (+ 34743us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:34.535189 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:34.535536 (+ 347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:34.536449 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:34.536454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:34.537207 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:34.537932 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:34.537947 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.540189 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.540192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:34.541502 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:34.541515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.543028 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.543037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:34.543229 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:34.547249 (+ 4020us) spinlock_profiling.cc:243] Waited 2.77 ms on lock 0x60c0000349f8. stack: 00007f891678166e 00007f8916781430 00007f89158085ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f89167770fd 00007f8916776d1e 00007f891d28e664 00007f891d18d810 00007f891d18a470 00007f891d18533a 00007f891d20cdca 00007f891d20af68 00007f891d23e82f 00007f891d23e58d
0809 04:50:34.548740 (+ 1491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:34.548766 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:34.553180 (+ 4414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:34.559436 (+ 6256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:34.559440 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:34.559444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:34.560088 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:34.560586 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:34.560591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:34.560595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:34.560799 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:34.561649 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:34.561653 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:34.561835 (+ 182us) client_negotiation.cc:769] Sending connection context
0809 04:50:34.562016 (+ 181us) client_negotiation.cc:240] Negotiation successful
0809 04:50:34.562333 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34597,"spinlock_wait_cycles":6375808,"thread_start_us":34425,"threads_started":1}
I20260809 04:50:34.564105 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:34.500881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:60995 (local address 127.31.93.62:45495)
0809 04:50:34.528050 (+ 27169us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:34.528057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:34.535421 (+ 7364us) server_negotiation.cc:407] Connection header received
0809 04:50:34.535698 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:34.535704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:34.535804 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:34.536095 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:34.538745 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.540054 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.541637 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.542414 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.542536 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:34.553402 (+ 10866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:34.553408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:34.553413 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:34.553452 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:34.559273 (+ 5821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:34.560230 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:34.560236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:34.560240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:34.560450 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:34.560942 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:34.560947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:34.560950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:34.561413 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:34.561581 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:34.563371 (+ 1790us) server_negotiation.cc:299] Negotiation successful
0809 04:50:34.563576 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26991,"thread_start_us":27190,"threads_started":1}
I20260809 04:50:34.564777 11280 heartbeater.cc:347] Connected to a master server at 127.31.93.60:35871
I20260809 04:50:34.565169 11280 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:34.565330 10217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:34.565855 11283 heartbeater.cc:347] Connected to a master server at 127.31.93.62:45495
I20260809 04:50:34.566066 11283 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:34.566987 11280 heartbeater.cc:511] Master 127.31.93.60:35871 requested a full tablet report, sending...
I20260809 04:50:34.567039 11283 heartbeater.cc:511] Master 127.31.93.62:45495 requested a full tablet report, sending...
I20260809 04:50:34.569150 9724 ts_manager.cc:194] Re-registered known tserver with Master: 67bceba2f21444d69fdc7d07a7242c3c (127.31.93.1:33309)
I20260809 04:50:34.570091 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:34.539752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:53713 (local address 127.31.93.61:41063)
0809 04:50:34.548479 (+ 8727us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:34.548485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:34.550832 (+ 2347us) server_negotiation.cc:407] Connection header received
0809 04:50:34.550897 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:34.550903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:34.550991 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:34.551242 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:34.552316 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.553937 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.556172 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.556967 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.557078 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:34.561457 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:34.561463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:34.561467 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:34.561504 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:34.565885 (+ 4381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:34.567299 (+ 1414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:34.567342 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:34.567345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:34.567486 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:34.568232 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:34.568236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:34.568239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:34.568563 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:34.568692 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:34.569383 (+ 691us) server_negotiation.cc:299] Negotiation successful
0809 04:50:34.569623 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8596,"thread_start_us":8469,"threads_started":1}
I20260809 04:50:34.570205 11004 ts_manager.cc:194] Registered new tserver with Master: 67bceba2f21444d69fdc7d07a7242c3c (127.31.93.1:33309)
I20260809 04:50:34.570470 9724 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:60995
I20260809 04:50:34.570765 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:34.539899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:41063 (local address 127.31.93.1:53713)
0809 04:50:34.550395 (+ 10496us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:34.550413 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:34.550720 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:34.551402 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:34.551407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:34.551857 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:34.552152 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:34.552161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.554117 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.554122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:34.556052 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:34.556065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:34.557245 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:34.557253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:34.557366 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:34.557903 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:34.557927 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:34.561243 (+ 3316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:34.566033 (+ 4790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:34.566038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:34.566041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:34.567155 (+ 1114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:34.567675 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:34.567680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:34.567683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:34.568096 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:34.568764 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:34.568768 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:34.568892 (+ 124us) client_negotiation.cc:769] Sending connection context
0809 04:50:34.569026 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 04:50:34.570429 (+ 1403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10279,"thread_start_us":9174,"threads_started":1}
I20260809 04:50:34.572172 11285 heartbeater.cc:347] Connected to a master server at 127.31.93.61:41063
I20260809 04:50:34.572424 11285 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:34.573115 11285 heartbeater.cc:511] Master 127.31.93.61:41063 requested a full tablet report, sending...
I20260809 04:50:34.575045 10687 ts_manager.cc:194] Registered new tserver with Master: 67bceba2f21444d69fdc7d07a7242c3c (127.31.93.1:33309)
I20260809 04:50:34.655068 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 10023
I20260809 04:50:34.677357 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:40459
--local_ip_for_outbound_sockets=127.31.93.2
--tserver_master_addrs=127.31.93.62:45495
--webserver_port=35897
--webserver_interface=127.31.93.2
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.2.keytab
--principal=oryx/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.93.62:45495,127.31.93.61:41063
--tserver_master_addrs=127.31.93.62:45495,127.31.93.61:41063,127.31.93.60:35871 with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:35.009212 11295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:35.009512 11295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:35.009670 11295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:35.016844 11295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:35.016952 11295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:35.017109 11295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:50:35.030995 11295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.2.keytab
--principal=oryx/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:40459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=35897
--tserver_master_addrs=127.31.93.62:45495,127.31.93.61:41063,127.31.93.60:35871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:35.032598 11295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:35.034276 11295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:35.048236 11303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:35.048532 11304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:35.048780 11307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:35.049546 11295 server_base.cc:1034] running on GCE node
I20260809 04:50:35.050153 11295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:35.051101 11295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:35.052318 11295 hybrid_clock.cc:648] HybridClock initialized: now 1786251035052259 us; error 44 us; skew 500 ppm
I20260809 04:50:35.052632 11295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:35 dist-test-slave-4tdj krb5kdc[9655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251035, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:35.055791 11295 init.cc:377] Logged in from keytab as oryx/127.31.93.2@KRBTEST.COM (short username oryx)
I20260809 04:50:35.057139 11295 webserver.cc:460] Webserver started at http://127.31.93.2:35897/ using document root <none> and password file <none>
I20260809 04:50:35.057755 11295 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:35.057868 11295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:35.063118 11295 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 04:50:35.066020 11314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:35.067405 11295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:50:35.067564 11295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "3480182938594d0a84d8c3986d8ee6be"
format_stamp: "Formatted at 2026-08-09 04:50:31 on dist-test-slave-4tdj"
server_key: "b27569ad7ca0bf84f8cf841cb1b9e3bd"
server_key_iv: "2438b0547b0c37c818dd694bbdb1699d"
server_key_version: "encryptionkey@0"
I20260809 04:50:35.068468 11295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:35.094983 11295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:35.099167 11295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:35.099428 11295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:35.100445 11295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:35.101778 11295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:35.101864 11295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:35.101962 11295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:35.102034 11295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:35.135596 11295 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:40459
I20260809 04:50:35.135651 11488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:40459 every 8 connection(s)
I20260809 04:50:35.136955 11295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:50:35.138034 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 11295
I20260809 04:50:35.138392 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 10233
Aug 09 04:50:35 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.93.3@KRBTEST.COM
I20260809 04:50:35.175809 11497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.139530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.31.93.3:43813)
0809 04:50:35.154083 (+ 14553us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:35.154090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:35.154116 (+ 26us) server_negotiation.cc:407] Connection header received
0809 04:50:35.154194 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:35.154202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:35.154483 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:35.154632 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:35.156741 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.158497 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.160518 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.161403 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.162160 (+ 757us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.166890 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:35.166905 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:35.166923 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:35.166970 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:35.171212 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.172194 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.172203 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.172206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.172460 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.173120 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.173125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.173127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.173566 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:35.174896 (+ 1330us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:35.175047 (+ 151us) server_negotiation.cc:299] Negotiation successful
0809 04:50:35.175247 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14383,"thread_start_us":14187,"threads_started":1}
I20260809 04:50:35.177340 10446 generic_service.cc:196] Checking for leaks (request via RPC)
Aug 09 04:50:35 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251035, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.61@KRBTEST.COM
Aug 09 04:50:35 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251035, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.60@KRBTEST.COM
Aug 09 04:50:35 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251035, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM
I20260809 04:50:35.196106 11499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.139894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:37475 (local address 127.31.93.60:35871)
0809 04:50:35.170224 (+ 30330us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:35.170230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:35.170251 (+ 21us) server_negotiation.cc:407] Connection header received
0809 04:50:35.170531 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:35.170536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:35.170613 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:35.170776 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:35.174446 (+ 3670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.175755 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.179281 (+ 3526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.180066 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.180177 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.187419 (+ 7242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:35.187426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:35.187432 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:35.187469 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:35.191937 (+ 4468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.193191 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.193197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.193201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.193350 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.193873 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.193879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.193882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.194278 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:35.194452 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:35.195464 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0809 04:50:35.195701 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30189,"thread_start_us":30389,"threads_started":1}
I20260809 04:50:35.196285 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.155125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:38201 (local address 127.31.93.62:45495)
0809 04:50:35.172320 (+ 17195us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:35.172325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:35.172344 (+ 19us) server_negotiation.cc:407] Connection header received
0809 04:50:35.172525 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:35.172568 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:35.172652 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:35.172864 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:35.176066 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.177098 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.179512 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.180137 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.180255 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.187927 (+ 7672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:35.187933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:35.187937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:35.187970 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:35.192725 (+ 4755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.193705 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.193711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.193714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.193864 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.194403 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.194408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.194411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.194858 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:35.194994 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:35.195536 (+ 542us) server_negotiation.cc:299] Negotiation successful
0809 04:50:35.195798 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17080,"thread_start_us":17320,"threads_started":1}
I20260809 04:50:35.197950 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.140483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:41063 (local address 127.31.93.2:52561)
0809 04:50:35.163816 (+ 23333us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:35.163872 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:35.169552 (+ 5680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:35.170595 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:35.170607 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:35.171607 (+ 1000us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:35.173758 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.173784 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.175577 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.175581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.178191 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.178205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.180946 (+ 2741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.180953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.181216 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.182261 (+ 1045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:35.182319 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:35.189291 (+ 6972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:35.193547 (+ 4256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:35.193551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:35.193555 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:35.194045 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:35.194520 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:35.194524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:35.194527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:35.194715 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:35.195530 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:35.195534 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:35.195713 (+ 179us) client_negotiation.cc:769] Sending connection context
0809 04:50:35.195869 (+ 156us) client_negotiation.cc:240] Negotiation successful
0809 04:50:35.196412 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23017,"mutex_wait_us":3,"thread_start_us":22770,"threads_started":1}
I20260809 04:50:35.198141 11502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.141160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:35871 (local address 127.31.93.2:37475)
0809 04:50:35.168548 (+ 27388us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:35.168570 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:35.170385 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:35.170984 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:35.170988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:35.172825 (+ 1837us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:35.174224 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.174239 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.176876 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.176881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.179096 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.179112 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.180531 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.180538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.180916 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.181946 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:35.181980 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:35.187189 (+ 5209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:35.192123 (+ 4934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:35.192132 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:35.192149 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:35.193048 (+ 899us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:35.193498 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:35.193503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:35.193506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:35.193734 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:35.194547 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:35.194555 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:35.194964 (+ 409us) client_negotiation.cc:769] Sending connection context
0809 04:50:35.196359 (+ 1395us) client_negotiation.cc:240] Negotiation successful
0809 04:50:35.196637 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27214,"thread_start_us":27016,"threads_started":1}
I20260809 04:50:35.198665 11508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.154883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.2:38201)
0809 04:50:35.169361 (+ 14478us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:35.169377 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:35.169551 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:35.172916 (+ 3365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:35.172921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:35.174229 (+ 1308us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:35.174826 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.174841 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.177300 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.177308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.179367 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.179383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.180412 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.180422 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.180730 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.184239 (+ 3509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:35.184261 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:35.187549 (+ 3288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:35.192883 (+ 5334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:35.192888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:35.192892 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:35.193565 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:35.194022 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:35.194026 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:35.194029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:35.194255 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:35.195080 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:35.195084 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:35.195224 (+ 140us) client_negotiation.cc:769] Sending connection context
0809 04:50:35.196531 (+ 1307us) client_negotiation.cc:240] Negotiation successful
0809 04:50:35.196773 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14365,"mutex_wait_us":328,"thread_start_us":14159,"threads_started":1}
I20260809 04:50:35.199344 11501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.141087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:52561 (local address 127.31.93.61:41063)
0809 04:50:35.167937 (+ 26850us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:35.167944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:35.167968 (+ 24us) server_negotiation.cc:407] Connection header received
0809 04:50:35.169958 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:35.169964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:35.170050 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:35.171801 (+ 1751us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:35.173946 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.175339 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.178614 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.180543 (+ 1929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.180659 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.189590 (+ 8931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:35.189597 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:35.189602 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:35.189644 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:35.193373 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.194186 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.194191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.194194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.194381 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.194853 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.194857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.194860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.195322 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:35.195450 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:35.198740 (+ 3290us) server_negotiation.cc:299] Negotiation successful
0809 04:50:35.198961 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26715,"thread_start_us":27699,"threads_started":1}
I20260809 04:50:35.200519 11493 heartbeater.cc:347] Connected to a master server at 127.31.93.61:41063
I20260809 04:50:35.200664 11492 heartbeater.cc:347] Connected to a master server at 127.31.93.62:45495
I20260809 04:50:35.200925 11493 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:35.200948 11492 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:35.201139 11491 heartbeater.cc:347] Connected to a master server at 127.31.93.60:35871
I20260809 04:50:35.201277 11491 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:35.202823 11492 heartbeater.cc:511] Master 127.31.93.62:45495 requested a full tablet report, sending...
I20260809 04:50:35.203166 11493 heartbeater.cc:511] Master 127.31.93.61:41063 requested a full tablet report, sending...
I20260809 04:50:35.203464 11491 heartbeater.cc:511] Master 127.31.93.60:35871 requested a full tablet report, sending...
I20260809 04:50:35.205119 9724 ts_manager.cc:194] Re-registered known tserver with Master: 3480182938594d0a84d8c3986d8ee6be (127.31.93.2:40459)
I20260809 04:50:35.205152 10687 ts_manager.cc:194] Registered new tserver with Master: 3480182938594d0a84d8c3986d8ee6be (127.31.93.2:40459)
I20260809 04:50:35.206235 11004 ts_manager.cc:194] Registered new tserver with Master: 3480182938594d0a84d8c3986d8ee6be (127.31.93.2:40459)
I20260809 04:50:35.206218 9724 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:38201
I20260809 04:50:35.272672 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 10233
I20260809 04:50:35.292701 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:43813
--local_ip_for_outbound_sockets=127.31.93.3
--tserver_master_addrs=127.31.93.62:45495
--webserver_port=32855
--webserver_interface=127.31.93.3
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.3.keytab
--principal=oryx/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.93.62:45495,127.31.93.61:41063
--tserver_master_addrs=127.31.93.62:45495,127.31.93.61:41063,127.31.93.60:35871 with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:50:35.573285 11283 heartbeater.cc:503] Master 127.31.93.62:45495 was elected leader, sending a full tablet report...
W20260809 04:50:35.610355 11515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:35.610610 11515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:35.610713 11515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:35.617584 11515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:35.617681 11515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:35.617806 11515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:50:35.626986 11515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.3.keytab
--principal=oryx/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:43813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=32855
--tserver_master_addrs=127.31.93.62:45495,127.31.93.61:41063,127.31.93.60:35871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:35.628206 11515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:35.629482 11515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:35.641861 11520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:35.642310 11523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:35.642510 11521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:35.643649 11515 server_base.cc:1034] running on GCE node
I20260809 04:50:35.644330 11515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:35.645218 11515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:35.646399 11515 hybrid_clock.cc:648] HybridClock initialized: now 1786251035646344 us; error 47 us; skew 500 ppm
I20260809 04:50:35.646673 11515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:35 dist-test-slave-4tdj krb5kdc[9655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251035, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:35.649624 11515 init.cc:377] Logged in from keytab as oryx/127.31.93.3@KRBTEST.COM (short username oryx)
I20260809 04:50:35.651021 11515 webserver.cc:460] Webserver started at http://127.31.93.3:32855/ using document root <none> and password file <none>
I20260809 04:50:35.651721 11515 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:35.651865 11515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:35.657075 11515 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260809 04:50:35.659626 11535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:35.660952 11515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 04:50:35.661106 11515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "c5ecb8b991eb4fceae5fb19e581391ee"
format_stamp: "Formatted at 2026-08-09 04:50:32 on dist-test-slave-4tdj"
server_key: "fae77142420c4212f3751eb826a02c39"
server_key_iv: "7a15a382ceeaf33daa71770bca4d4232"
server_key_version: "encryptionkey@0"
I20260809 04:50:35.662075 11515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:35.680842 11515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:35.684674 11515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:35.684912 11515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:35.685832 11515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:35.687161 11515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:35.687258 11515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:35.687368 11515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:35.687446 11515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:35.724552 11515 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:43813
I20260809 04:50:35.724627 11727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:43813 every 8 connection(s)
I20260809 04:50:35.725993 11515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:50:35.733067 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 11515
Aug 09 04:50:35 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251035, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.60@KRBTEST.COM
I20260809 04:50:35.760722 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.729610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:46915 (local address 127.31.93.60:35871)
0809 04:50:35.730410 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:35.730416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:35.734285 (+ 3869us) server_negotiation.cc:407] Connection header received
0809 04:50:35.734379 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:35.734384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:35.734464 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:35.734569 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:35.737222 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.738257 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.745875 (+ 7618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.746571 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.746698 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.751852 (+ 5154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:35.751859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:35.751865 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:35.751924 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:35.756573 (+ 4649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.757620 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.757626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.757629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.757809 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.758521 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.758527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.758531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.758972 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:35.759107 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:35.760075 (+ 968us) server_negotiation.cc:299] Negotiation successful
0809 04:50:35.760323 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":562,"threads_started":1}
Aug 09 04:50:35 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251035, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM
I20260809 04:50:35.769773 11736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.729867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:35871 (local address 127.31.93.3:46915)
0809 04:50:35.734015 (+ 4148us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:35.734033 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:35.734164 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:35.734962 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:35.734974 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:35.736121 (+ 1147us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:35.737034 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.737051 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.743931 (+ 6880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.743939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.745691 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.745709 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.747063 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.747072 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.747361 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.748241 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:35.748272 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:35.751571 (+ 3299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:35.756773 (+ 5202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:35.756785 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:35.756802 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:35.757479 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:35.757971 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:35.757975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:35.757979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:35.758396 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:35.759214 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:35.759221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:35.759613 (+ 392us) client_negotiation.cc:769] Sending connection context
0809 04:50:35.766149 (+ 6536us) client_negotiation.cc:240] Negotiation successful
0809 04:50:35.767301 (+ 1152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3998,"mutex_wait_us":567,"thread_start_us":920,"threads_started":1}
I20260809 04:50:35.771644 11728 heartbeater.cc:347] Connected to a master server at 127.31.93.60:35871
I20260809 04:50:35.772075 11728 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:35.773254 11728 heartbeater.cc:511] Master 127.31.93.60:35871 requested a full tablet report, sending...
Aug 09 04:50:35 dist-test-slave-4tdj krb5kdc[9655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251035, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.61@KRBTEST.COM
I20260809 04:50:35.776178 11004 ts_manager.cc:194] Registered new tserver with Master: c5ecb8b991eb4fceae5fb19e581391ee (127.31.93.3:43813)
I20260809 04:50:35.784228 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.729535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.3:48707)
0809 04:50:35.731114 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:35.731160 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:35.733243 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:35.734963 (+ 1720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:35.734974 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:35.736111 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:35.737026 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.737052 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.738827 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.738832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.740926 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.740943 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.761078 (+ 20135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.761086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.761374 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.762151 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:35.762175 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:35.766863 (+ 4688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:35.780778 (+ 13915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:35.780784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:35.780787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:35.781248 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:35.781710 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:35.781716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:35.781719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:35.781926 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:35.783041 (+ 1115us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:35.783046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:35.783200 (+ 154us) client_negotiation.cc:769] Sending connection context
0809 04:50:35.783337 (+ 137us) client_negotiation.cc:240] Negotiation successful
0809 04:50:35.783755 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1333,"mutex_wait_us":62,"spinlock_wait_cycles":1738496,"thread_start_us":4028,"threads_started":1}
I20260809 04:50:35.785342 11734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.729331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:48707 (local address 127.31.93.62:45495)
0809 04:50:35.730058 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:35.730065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:35.731273 (+ 1208us) server_negotiation.cc:407] Connection header received
0809 04:50:35.734389 (+ 3116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:35.734395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:35.734485 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:35.734701 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:35.737488 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.738630 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.759259 (+ 20629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.760070 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.760193 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.767322 (+ 7129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:35.767331 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:35.767336 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:35.767377 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:35.780626 (+ 13249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.781425 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.781431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.781434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.781577 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.782098 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.782103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.782106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.782905 (+ 799us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:35.784569 (+ 1664us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:35.784686 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 04:50:35.784921 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":534,"threads_started":1}
I20260809 04:50:35.786247 11729 heartbeater.cc:347] Connected to a master server at 127.31.93.62:45495
I20260809 04:50:35.786420 11729 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:35.786645 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.729032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:60257 (local address 127.31.93.61:41063)
0809 04:50:35.729732 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:35.729739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:35.734492 (+ 4753us) server_negotiation.cc:407] Connection header received
0809 04:50:35.734741 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:35.734746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:35.734825 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:35.734904 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:35.737467 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.738630 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.763603 (+ 24973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.764590 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.764694 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.777897 (+ 13203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:35.777904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:35.777908 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:35.777951 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:35.782752 (+ 4801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.784119 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.784125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.784128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.784292 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.784773 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.784778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.784782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.785102 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:35.785244 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:35.786020 (+ 776us) server_negotiation.cc:299] Negotiation successful
0809 04:50:35.786237 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":757,"threads_started":1}
I20260809 04:50:35.787040 11729 heartbeater.cc:511] Master 127.31.93.62:45495 requested a full tablet report, sending...
I20260809 04:50:35.787602 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.729535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:41063 (local address 127.31.93.3:60257)
0809 04:50:35.734390 (+ 4855us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:35.734406 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:35.734588 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:35.735124 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:35.735128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:35.736310 (+ 1182us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:35.737324 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.737333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.738826 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.738832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.758366 (+ 19534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.758381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.765341 (+ 6960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.765350 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.765569 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.768482 (+ 2913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:35.768516 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:35.776334 (+ 7818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:35.782999 (+ 6665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:35.783004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:35.783007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:35.783666 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:35.784435 (+ 769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:35.784440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:35.784443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:35.784600 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:35.785329 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:35.785333 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:35.785504 (+ 171us) client_negotiation.cc:769] Sending connection context
0809 04:50:35.786821 (+ 1317us) client_negotiation.cc:240] Negotiation successful
0809 04:50:35.787117 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4709,"mutex_wait_us":18707,"thread_start_us":3986,"threads_started":1}
I20260809 04:50:35.788511 9724 ts_manager.cc:194] Re-registered known tserver with Master: c5ecb8b991eb4fceae5fb19e581391ee (127.31.93.3:43813)
I20260809 04:50:35.789145 11730 heartbeater.cc:347] Connected to a master server at 127.31.93.61:41063
I20260809 04:50:35.789275 11730 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:35.789812 9724 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:48707
I20260809 04:50:35.789884 11730 heartbeater.cc:511] Master 127.31.93.61:41063 requested a full tablet report, sending...
I20260809 04:50:35.791204 10687 ts_manager.cc:194] Registered new tserver with Master: c5ecb8b991eb4fceae5fb19e581391ee (127.31.93.3:43813)
I20260809 04:50:35.803471 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.739826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53138 (local address 127.31.93.61:41063)
0809 04:50:35.780861 (+ 41035us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:35.780869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:35.780894 (+ 25us) server_negotiation.cc:407] Connection header received
0809 04:50:35.780956 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:35.780967 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:35.781063 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:35.781191 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:35.787283 (+ 6092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.788502 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.790669 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.791827 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.791959 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.796546 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:35.796555 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:35.796559 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:35.796600 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:35.800004 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.800840 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.800845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.800848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.801005 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.801443 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.801449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.801453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.801928 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:35.802180 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:35.802827 (+ 647us) server_negotiation.cc:299] Negotiation successful
0809 04:50:35.803048 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40854,"thread_start_us":45931,"threads_started":1}
I20260809 04:50:35.807317 11750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.738614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.31.93.62:45495)
0809 04:50:35.785311 (+ 46697us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:35.785319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:35.785347 (+ 28us) server_negotiation.cc:407] Connection header received
0809 04:50:35.785723 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:35.785736 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:35.785869 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:35.790337 (+ 4468us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:35.792397 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.793627 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.796048 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.796866 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.796977 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.800921 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:35.800927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:35.800932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:35.800967 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:35.804613 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.805417 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.805420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.805422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.805548 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.805969 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.805974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.805978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.806228 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:35.806588 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:35.806853 (+ 265us) server_negotiation.cc:299] Negotiation successful
0809 04:50:35.807064 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46548,"thread_start_us":46997,"threads_started":1}
I20260809 04:50:35.808840 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.750552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.31.93.60:35871)
0809 04:50:35.764976 (+ 14424us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:35.764982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:35.787851 (+ 22869us) server_negotiation.cc:407] Connection header received
0809 04:50:35.788127 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:35.788132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:35.788249 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:35.788318 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:35.793866 (+ 5548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.794768 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.796536 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.797339 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.797462 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.801336 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:35.801343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:35.801347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:35.801385 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:35.805431 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.806173 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.806180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.806183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.806349 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:35.806757 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:35.806762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:35.806766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:35.807013 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:35.807261 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:35.808386 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0809 04:50:35.808557 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14250,"thread_start_us":26510,"threads_started":1}
I20260809 04:50:35.811370 9724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:50:35.820290 9724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:50:35.865259 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.843095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:40459 (local address 127.0.0.1:51464)
0809 04:50:35.855768 (+ 12673us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:35.855786 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:35.855925 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:35.856936 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:35.856940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:35.856964 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:35.857379 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.857394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.860016 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.860021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.862264 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.862280 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.864357 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.864365 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.864614 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.864639 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 04:50:35.864724 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 04:50:35.864847 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12520,"thread_start_us":12930,"threads_started":1}
I20260809 04:50:35.865365 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.843354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.31.93.2:40459)
0809 04:50:35.843909 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:35.843919 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:35.855893 (+ 11974us) server_negotiation.cc:407] Connection header received
0809 04:50:35.856141 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:35.856150 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:35.856469 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:35.856754 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:35.858395 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.859840 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.862432 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.864073 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.864191 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.864474 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:35.864789 (+ 315us) server_negotiation.cc:299] Negotiation successful
0809 04:50:35.864987 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":292,"threads_started":1}
I20260809 04:50:35.867241 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.851413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:33309 (local address 127.0.0.1:37398)
0809 04:50:35.856608 (+ 5195us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:35.856651 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:35.856810 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:35.858184 (+ 1374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:35.858188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:35.858215 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:35.858988 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.859004 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.861209 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.861213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.863453 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.863472 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.866514 (+ 3042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.866519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.866641 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.866664 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:50:35.866734 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 04:50:35.866852 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5049,"thread_start_us":4884,"threads_started":1}
I20260809 04:50:35.867362 11782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.851781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.31.93.1:33309)
0809 04:50:35.852476 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:35.852486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:35.857547 (+ 5061us) server_negotiation.cc:407] Connection header received
0809 04:50:35.857644 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:35.857650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:35.857939 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:35.858188 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:35.859216 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.861059 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.863620 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.865994 (+ 2374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.866102 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.866372 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:35.866819 (+ 447us) server_negotiation.cc:299] Negotiation successful
0809 04:50:35.866984 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":343,"threads_started":1}
I20260809 04:50:35.869562 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.840547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:43813 (local address 127.0.0.1:59582)
0809 04:50:35.843040 (+ 2493us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:35.843059 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:35.843236 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:35.857803 (+ 14567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:35.857807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:35.857832 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:35.858745 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.858758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.863308 (+ 4550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.863313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.865801 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:35.865818 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.868537 (+ 2719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.868546 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:35.868831 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.868867 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 04:50:35.868965 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 04:50:35.869089 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2341,"thread_start_us":2119,"threads_started":1}
I20260809 04:50:35.869578 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:35.840872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.31.93.3:43813)
0809 04:50:35.857017 (+ 16145us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:35.857026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:35.857073 (+ 47us) server_negotiation.cc:407] Connection header received
0809 04:50:35.857199 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:35.857206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:35.857434 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:35.857707 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:35.859055 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.863175 (+ 4120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.865989 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:35.868195 (+ 2206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:35.868333 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:35.868541 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:35.869031 (+ 490us) server_negotiation.cc:299] Negotiation successful
0809 04:50:35.869195 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15938,"thread_start_us":17149,"threads_started":1}
I20260809 04:50:35.870345 11394 tablet_service.cc:1467] Processing CreateTablet for tablet 94f1c1cf694048a8b26af620c63aaaf6 (DEFAULT_TABLE table=test-table [id=b92542a680ca48df8f7843430f83341f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:35.871232 11168 tablet_service.cc:1467] Processing CreateTablet for tablet 94f1c1cf694048a8b26af620c63aaaf6 (DEFAULT_TABLE table=test-table [id=b92542a680ca48df8f7843430f83341f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:35.871850 11394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f1c1cf694048a8b26af620c63aaaf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:35.872457 11168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f1c1cf694048a8b26af620c63aaaf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:35.878445 11610 tablet_service.cc:1467] Processing CreateTablet for tablet 94f1c1cf694048a8b26af620c63aaaf6 (DEFAULT_TABLE table=test-table [id=b92542a680ca48df8f7843430f83341f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:35.880141 11610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f1c1cf694048a8b26af620c63aaaf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:35.886396 11785 tablet_bootstrap.cc:492] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be: Bootstrap starting.
I20260809 04:50:35.889492 11788 tablet_bootstrap.cc:492] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c: Bootstrap starting.
I20260809 04:50:35.891971 11785 tablet_bootstrap.cc:654] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:35.893251 11785 log.cc:824] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:35.893339 11788 tablet_bootstrap.cc:654] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:35.895133 11788 log.cc:824] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:35.896638 11785 tablet_bootstrap.cc:492] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be: No bootstrap required, opened a new log
I20260809 04:50:35.896978 11785 ts_tablet_manager.cc:1397] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260809 04:50:35.897742 11791 tablet_bootstrap.cc:492] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee: Bootstrap starting.
I20260809 04:50:35.897934 11788 tablet_bootstrap.cc:492] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c: No bootstrap required, opened a new log
I20260809 04:50:35.898232 11788 ts_tablet_manager.cc:1397] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260809 04:50:35.902874 11791 tablet_bootstrap.cc:654] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:35.904271 11791 log.cc:824] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:35.906984 11785 raft_consensus.cc:348] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } }
I20260809 04:50:35.907392 11785 raft_consensus.cc:374] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:35.907487 11785 raft_consensus.cc:729] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3480182938594d0a84d8c3986d8ee6be, State: Initialized, Role: FOLLOWER
I20260809 04:50:35.908082 11785 consensus_queue.cc:260] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } }
I20260809 04:50:35.908742 11791 tablet_bootstrap.cc:492] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee: No bootstrap required, opened a new log
I20260809 04:50:35.909085 11791 ts_tablet_manager.cc:1397] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260809 04:50:35.909934 11788 raft_consensus.cc:348] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } }
I20260809 04:50:35.910503 11788 raft_consensus.cc:374] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:35.910676 11788 raft_consensus.cc:729] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67bceba2f21444d69fdc7d07a7242c3c, State: Initialized, Role: FOLLOWER
I20260809 04:50:35.911368 11788 consensus_queue.cc:260] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } }
I20260809 04:50:35.913833 11788 ts_tablet_manager.cc:1428] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
I20260809 04:50:35.914974 11492 heartbeater.cc:503] Master 127.31.93.62:45495 was elected leader, sending a full tablet report...
I20260809 04:50:35.915421 11785 ts_tablet_manager.cc:1428] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be: Time spent starting tablet: real 0.018s user 0.011s sys 0.009s
I20260809 04:50:35.926618 11791 raft_consensus.cc:348] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } }
I20260809 04:50:35.927101 11791 raft_consensus.cc:374] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:35.927192 11791 raft_consensus.cc:729] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ecb8b991eb4fceae5fb19e581391ee, State: Initialized, Role: FOLLOWER
I20260809 04:50:35.927704 11791 consensus_queue.cc:260] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } }
I20260809 04:50:35.929183 11729 heartbeater.cc:503] Master 127.31.93.62:45495 was elected leader, sending a full tablet report...
I20260809 04:50:35.929620 11791 ts_tablet_manager.cc:1428] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee: Time spent starting tablet: real 0.020s user 0.011s sys 0.005s
W20260809 04:50:35.981154 11731 tablet.cc:2369] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:50:36.020882 11795 raft_consensus.cc:482] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:50:36.021201 11795 raft_consensus.cc:504] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } }
I20260809 04:50:36.023342 11795 leader_election.cc:288] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ecb8b991eb4fceae5fb19e581391ee (127.31.93.3:43813), 3480182938594d0a84d8c3986d8ee6be (127.31.93.2:40459)
I20260809 04:50:36.032464 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:36.023667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:54601 (local address 127.31.93.3:43813)
0809 04:50:36.023900 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:36.023908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:36.024712 (+ 804us) server_negotiation.cc:407] Connection header received
0809 04:50:36.025023 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:36.025029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:36.025101 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:36.025157 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:36.026335 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.027316 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.030119 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.031366 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.031481 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:36.031577 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:36.032087 (+ 510us) server_negotiation.cc:299] Negotiation successful
0809 04:50:36.032227 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:50:36.033326 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:36.023546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:43813 (local address 127.31.93.1:54601)
0809 04:50:36.024563 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:36.024584 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:36.024840 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:36.025365 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:36.025370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:36.025399 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:36.026171 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:36.026185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.027439 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.027444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:36.029943 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:36.029957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.031663 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.031670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:36.031868 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:36.031906 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 04:50:36.032647 (+ 741us) client_negotiation.cc:240] Negotiation successful
0809 04:50:36.032775 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":2,"thread_start_us":647,"threads_started":1}
I20260809 04:50:36.033983 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:36.024739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:33253 (local address 127.31.93.2:40459)
0809 04:50:36.024954 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:36.024960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:36.025776 (+ 816us) server_negotiation.cc:407] Connection header received
0809 04:50:36.025973 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:36.025979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:36.026042 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:36.026123 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:36.026894 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.028006 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.030897 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.032264 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.032386 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:36.032493 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:36.033539 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0809 04:50:36.033671 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 04:50:36.034549 11647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f1c1cf694048a8b26af620c63aaaf6" candidate_uuid: "67bceba2f21444d69fdc7d07a7242c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" is_pre_election: true
I20260809 04:50:36.034968 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:36.024607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:40459 (local address 127.31.93.1:33253)
0809 04:50:36.025655 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:36.025670 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:36.025833 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:36.026160 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:36.026164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:36.026187 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:36.026725 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:36.026739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.028152 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.028157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:36.030744 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:36.030759 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.033101 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.033109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:36.033344 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:36.033379 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 04:50:36.034273 (+ 894us) client_negotiation.cc:240] Negotiation successful
0809 04:50:36.034425 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":27,"thread_start_us":609,"threads_started":1}
I20260809 04:50:36.035231 11647 raft_consensus.cc:2454] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67bceba2f21444d69fdc7d07a7242c3c in term 0.
I20260809 04:50:36.035938 11424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f1c1cf694048a8b26af620c63aaaf6" candidate_uuid: "67bceba2f21444d69fdc7d07a7242c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3480182938594d0a84d8c3986d8ee6be" is_pre_election: true
I20260809 04:50:36.036064 11102 leader_election.cc:302] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67bceba2f21444d69fdc7d07a7242c3c, c5ecb8b991eb4fceae5fb19e581391ee; no voters:
I20260809 04:50:36.036414 11795 raft_consensus.cc:2781] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:50:36.036437 11424 raft_consensus.cc:2454] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67bceba2f21444d69fdc7d07a7242c3c in term 0.
I20260809 04:50:36.036535 11795 raft_consensus.cc:482] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:50:36.036641 11795 raft_consensus.cc:3037] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:36.039347 11795 raft_consensus.cc:504] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } }
I20260809 04:50:36.039913 11795 leader_election.cc:288] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [CANDIDATE]: Term 1 election: Requested vote from peers c5ecb8b991eb4fceae5fb19e581391ee (127.31.93.3:43813), 3480182938594d0a84d8c3986d8ee6be (127.31.93.2:40459)
I20260809 04:50:36.040336 11647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f1c1cf694048a8b26af620c63aaaf6" candidate_uuid: "67bceba2f21444d69fdc7d07a7242c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ecb8b991eb4fceae5fb19e581391ee"
I20260809 04:50:36.040490 11647 raft_consensus.cc:3037] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:36.040453 11424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f1c1cf694048a8b26af620c63aaaf6" candidate_uuid: "67bceba2f21444d69fdc7d07a7242c3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3480182938594d0a84d8c3986d8ee6be"
I20260809 04:50:36.040647 11424 raft_consensus.cc:3037] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be [term 0 FOLLOWER]: Advancing to term 1
W20260809 04:50:36.042629 11291 tablet.cc:2369] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:50:36.042811 11647 raft_consensus.cc:2454] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67bceba2f21444d69fdc7d07a7242c3c in term 1.
I20260809 04:50:36.043241 11102 leader_election.cc:302] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67bceba2f21444d69fdc7d07a7242c3c, c5ecb8b991eb4fceae5fb19e581391ee; no voters:
I20260809 04:50:36.043555 11795 raft_consensus.cc:2781] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:36.044096 11795 raft_consensus.cc:686] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [term 1 LEADER]: Becoming Leader. State: Replica: 67bceba2f21444d69fdc7d07a7242c3c, State: Running, Role: LEADER
I20260809 04:50:36.044327 11424 raft_consensus.cc:2454] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67bceba2f21444d69fdc7d07a7242c3c in term 1.
I20260809 04:50:36.044783 11795 consensus_queue.cc:237] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } }
I20260809 04:50:36.049921 9724 catalog_manager.cc:5495] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 67bceba2f21444d69fdc7d07a7242c3c (127.31.93.1). New cstate: current_term: 1 leader_uuid: "67bceba2f21444d69fdc7d07a7242c3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260809 04:50:36.091720 11782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:36.085282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37400 (local address 127.31.93.1:33309)
0809 04:50:36.085498 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:36.085504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:36.085524 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:50:36.085694 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:36.085700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:36.085807 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:36.085955 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:50:36.086269 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.087654 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.089194 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.089833 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.089951 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:36.090445 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:50:36.091061 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:50:36.091221 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:36.091275 (+ 54us) server_negotiation.cc:299] Negotiation successful
0809 04:50:36.091356 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 04:50:36.101825 11424 raft_consensus.cc:1261] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be [term 1 FOLLOWER]: Refusing update from remote peer 67bceba2f21444d69fdc7d07a7242c3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:36.101832 11647 raft_consensus.cc:1261] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee [term 1 FOLLOWER]: Refusing update from remote peer 67bceba2f21444d69fdc7d07a7242c3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:36.102909 11807 consensus_queue.cc:1035] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:36.103425 11795 consensus_queue.cc:1035] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:36.119227 11813 mvcc.cc:205] Tried to move back new op lower bound from 7316484243861479424 to 7316484243643551744. Current Snapshot: MvccSnapshot[applied={T|T < 7316484243861479424}]
I20260809 04:50:36.125558 11811 mvcc.cc:205] Tried to move back new op lower bound from 7316484243861479424 to 7316484243643551744. Current Snapshot: MvccSnapshot[applied={T|T < 7316484243861479424}]
I20260809 04:50:36.134939 9724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:36.135290 9724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:36.144485 9724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 391a359744cb4913b6a7a3fdfaff318a: Sending DeleteTablet for 3 replicas of tablet 94f1c1cf694048a8b26af620c63aaaf6
I20260809 04:50:36.145676 11610 tablet_service.cc:1514] Processing DeleteTablet for tablet 94f1c1cf694048a8b26af620c63aaaf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:36 UTC) from {username='oryx', principal='oryx/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:59582
I20260809 04:50:36.146275 11394 tablet_service.cc:1514] Processing DeleteTablet for tablet 94f1c1cf694048a8b26af620c63aaaf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:36 UTC) from {username='oryx', principal='oryx/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:51464
I20260809 04:50:36.146973 11168 tablet_service.cc:1514] Processing DeleteTablet for tablet 94f1c1cf694048a8b26af620c63aaaf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:36 UTC) from {username='oryx', principal='oryx/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:37398
I20260809 04:50:36.147135 11826 tablet_replica.cc:330] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee: stopping tablet replica
I20260809 04:50:36.147596 11826 raft_consensus.cc:2229] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:36.148164 11826 raft_consensus.cc:2258] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:36.150053 11826 ts_tablet_manager.cc:1905] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:36.158752 11826 ts_tablet_manager.cc:1918] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:36.159006 11826 log.cc:1196] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/94f1c1cf694048a8b26af620c63aaaf6
I20260809 04:50:36.159564 11826 ts_tablet_manager.cc:1939] T 94f1c1cf694048a8b26af620c63aaaf6 P c5ecb8b991eb4fceae5fb19e581391ee: Deleting consensus metadata
I20260809 04:50:36.161269 9703 catalog_manager.cc:4841] TS c5ecb8b991eb4fceae5fb19e581391ee (127.31.93.3:43813): tablet 94f1c1cf694048a8b26af620c63aaaf6 (table test-table [id=b92542a680ca48df8f7843430f83341f]) successfully deleted
I20260809 04:50:36.165477 11828 tablet_replica.cc:330] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c: stopping tablet replica
I20260809 04:50:36.166055 11828 raft_consensus.cc:2229] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:50:36.167081 11828 raft_consensus.cc:2258] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:36.167517 11827 tablet_replica.cc:330] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be: stopping tablet replica
I20260809 04:50:36.168020 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 9676
I20260809 04:50:36.167975 11827 raft_consensus.cc:2229] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:36.168596 11827 raft_consensus.cc:2258] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:36.169272 9770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:36.170861 11828 ts_tablet_manager.cc:1905] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:36.172453 11827 ts_tablet_manager.cc:1905] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:36.184693 11827 ts_tablet_manager.cc:1918] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:36.184851 11827 log.cc:1196] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/94f1c1cf694048a8b26af620c63aaaf6
I20260809 04:50:36.185457 11827 ts_tablet_manager.cc:1939] T 94f1c1cf694048a8b26af620c63aaaf6 P 3480182938594d0a84d8c3986d8ee6be: Deleting consensus metadata
I20260809 04:50:36.186540 11828 ts_tablet_manager.cc:1918] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:36.186764 11828 log.cc:1196] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/94f1c1cf694048a8b26af620c63aaaf6
I20260809 04:50:36.186877 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:36.150673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53142 (local address 127.31.93.61:41063)
0809 04:50:36.151227 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:36.151233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:36.166354 (+ 15121us) server_negotiation.cc:407] Connection header received
0809 04:50:36.166560 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:36.166565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:36.166648 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:36.166748 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:36.171588 (+ 4840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.172674 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.174529 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.175130 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.175226 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:36.178873 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:36.178880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:36.178885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:36.178926 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:36.182563 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:36.183694 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:36.183699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:36.183703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:36.183857 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:36.184502 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:36.184507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:36.184511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:36.185000 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:36.185405 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:36.186248 (+ 843us) server_negotiation.cc:299] Negotiation successful
0809 04:50:36.186501 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406}
I20260809 04:50:36.187649 11828 ts_tablet_manager.cc:1939] T 94f1c1cf694048a8b26af620c63aaaf6 P 67bceba2f21444d69fdc7d07a7242c3c: Deleting consensus metadata
I20260809 04:50:36.204375 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:36.166533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.31.93.60:35871)
0809 04:50:36.166703 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:36.166709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:36.188055 (+ 21346us) server_negotiation.cc:407] Connection header received
0809 04:50:36.188126 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:36.188131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:36.188213 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:36.188310 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:36.190481 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.191666 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.193646 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.194282 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.194419 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:36.197449 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:36.197456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:36.197461 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:36.197503 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:36.201216 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:36.202182 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:36.202188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:36.202191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:36.202393 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:36.202829 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:36.202833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:36.202837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:36.203242 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:36.203506 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:36.203799 (+ 293us) server_negotiation.cc:299] Negotiation successful
0809 04:50:36.204015 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 04:50:36.306011 9703 catalog_manager.cc:4841] TS 3480182938594d0a84d8c3986d8ee6be (127.31.93.2:40459): tablet 94f1c1cf694048a8b26af620c63aaaf6 (table test-table [id=b92542a680ca48df8f7843430f83341f]) successfully deleted
I20260809 04:50:36.307464 9705 catalog_manager.cc:4841] TS 67bceba2f21444d69fdc7d07a7242c3c (127.31.93.1:33309): tablet 94f1c1cf694048a8b26af620c63aaaf6 (table test-table [id=b92542a680ca48df8f7843430f83341f]) successfully deleted
I20260809 04:50:36.308125 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 9676
I20260809 04:50:36.364464 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:36.341027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.31.93.61:41063)
0809 04:50:36.341223 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:36.341230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:36.346983 (+ 5753us) server_negotiation.cc:407] Connection header received
0809 04:50:36.347968 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:36.347974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:36.348074 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:36.348146 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:36.350417 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.351573 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.353764 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.354222 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.354359 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:36.357837 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:36.357845 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:36.357851 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:36.357897 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:36.361503 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:36.362517 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:36.362523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:36.362527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:36.362658 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:36.362996 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:36.363000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:36.363002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:36.363254 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:36.363477 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:36.363855 (+ 378us) server_negotiation.cc:299] Negotiation successful
0809 04:50:36.364073 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 04:50:36.364732 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:36.340813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.31.93.60:35871)
0809 04:50:36.344555 (+ 3742us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:36.344562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:36.347999 (+ 3437us) server_negotiation.cc:407] Connection header received
0809 04:50:36.348066 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:36.348071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:36.348144 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:36.348224 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:36.350507 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.351590 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.353709 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:36.354483 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:36.354623 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:36.358439 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:36.358444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:36.358447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:36.358499 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:36.361009 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:36.361638 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:36.361642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:36.361645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:36.361775 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:36.362212 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:36.362218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:36.362222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:36.362684 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:36.362954 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:36.364039 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0809 04:50:36.364281 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3559}
I20260809 04:50:37.123232 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.122421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.1:53037)
0809 04:50:37.122887 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.123002 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":234,"threads_started":1}
W20260809 04:50:37.123728 11283 heartbeater.cc:644] Failed to heartbeat to 127.31.93.62:45495 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
I20260809 04:50:37.133567 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.132419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.3:49803)
0809 04:50:37.133230 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.133369 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":242,"threads_started":1}
W20260809 04:50:37.134475 11729 heartbeater.cc:644] Failed to heartbeat to 127.31.93.62:45495 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
I20260809 04:50:37.135226 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.134124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.2:35229)
0809 04:50:37.134925 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.135049 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":480,"threads_started":1}
W20260809 04:50:37.135653 11492 heartbeater.cc:644] Failed to heartbeat to 127.31.93.62:45495 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
I20260809 04:50:37.643949 11847 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 391a359744cb4913b6a7a3fdfaff318a)
I20260809 04:50:37.644131 11850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 391a359744cb4913b6a7a3fdfaff318a)
I20260809 04:50:37.644176 11847 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } }
I20260809 04:50:37.644399 11850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } }
I20260809 04:50:37.645524 11847 leader_election.cc:288] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 391a359744cb4913b6a7a3fdfaff318a (127.31.93.62:45495), 025ce77beaad4885a83c296e1694006e (127.31.93.61:41063)
I20260809 04:50:37.646701 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.645599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.0.0.1:43124)
0809 04:50:37.646168 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.646396 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":222,"threads_started":1}
W20260809 04:50:37.646960 10978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
I20260809 04:50:37.647819 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.645681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.0.0.1:43126)
0809 04:50:37.647342 (+ 1661us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.647563 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1541,"threads_started":1}
I20260809 04:50:37.653043 11850 leader_election.cc:288] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 391a359744cb4913b6a7a3fdfaff318a (127.31.93.62:45495), e0326546ecbb435d8d754a858e366376 (127.31.93.60:35871)
W20260809 04:50:37.655903 10654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
I20260809 04:50:37.656250 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.655732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.0.0.1:43134)
0809 04:50:37.655879 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.655975 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 04:50:37.656522 10978 leader_election.cc:334] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 391a359744cb4913b6a7a3fdfaff318a (127.31.93.62:45495): Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
I20260809 04:50:37.658188 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.657653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.0.0.1:43136)
0809 04:50:37.657838 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.657993 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 04:50:37.658408 10654 leader_election.cc:334] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 391a359744cb4913b6a7a3fdfaff318a (127.31.93.62:45495): Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
I20260809 04:50:37.661437 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.647245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.31.93.60:35871)
0809 04:50:37.654738 (+ 7493us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:37.654745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:37.654765 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:50:37.654835 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:37.654841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:37.654922 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:37.655453 (+ 531us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:37.655683 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.657084 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.659279 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.660591 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.660734 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:37.660827 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:37.661092 (+ 265us) server_negotiation.cc:299] Negotiation successful
0809 04:50:37.661187 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":7359,"thread_start_us":7218,"threads_started":1}
I20260809 04:50:37.661517 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.646785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:35871 (local address 127.0.0.1:60270)
0809 04:50:37.648125 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.648143 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:37.648280 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:37.655096 (+ 6816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:37.655101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:37.655131 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:37.655334 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:37.655343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.657212 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.657216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:37.659135 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:37.659148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.660747 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.660752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:37.660942 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:37.660979 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 04:50:37.661090 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 04:50:37.661223 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"thread_start_us":575,"threads_started":1}
I20260809 04:50:37.662149 11022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "025ce77beaad4885a83c296e1694006e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e0326546ecbb435d8d754a858e366376" is_pre_election: true
I20260809 04:50:37.662566 11022 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 025ce77beaad4885a83c296e1694006e in term 1.
I20260809 04:50:37.663053 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.654987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53172 (local address 127.31.93.61:41063)
0809 04:50:37.655927 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:37.655933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:37.655951 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:50:37.656012 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:37.656016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:37.656082 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:37.656936 (+ 854us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:37.657197 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.658664 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.660850 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.662106 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.662196 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:37.662268 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:37.662633 (+ 365us) server_negotiation.cc:299] Negotiation successful
0809 04:50:37.662750 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":826,"thread_start_us":667,"threads_started":1}
I20260809 04:50:37.663093 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.645713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:41063 (local address 127.0.0.1:53172)
0809 04:50:37.646429 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.646447 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:37.646579 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:37.656245 (+ 9666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:37.656251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:37.656290 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:37.656760 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:37.656774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.658787 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.658791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:37.660707 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:37.660719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.662280 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.662286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:37.662476 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:37.662507 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 04:50:37.662612 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 04:50:37.662731 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":1,"thread_start_us":459,"threads_started":1}
I20260809 04:50:37.663108 10655 leader_election.cc:302] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 025ce77beaad4885a83c296e1694006e, e0326546ecbb435d8d754a858e366376; no voters: 391a359744cb4913b6a7a3fdfaff318a
I20260809 04:50:37.663515 11850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 04:50:37.663642 11850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 391a359744cb4913b6a7a3fdfaff318a)
I20260809 04:50:37.663743 11850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 1 FOLLOWER]: Advancing to term 2
I20260809 04:50:37.665961 11850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } }
I20260809 04:50:37.666399 10710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0326546ecbb435d8d754a858e366376" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "025ce77beaad4885a83c296e1694006e" is_pre_election: true
I20260809 04:50:37.666649 11850 leader_election.cc:288] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [CANDIDATE]: Term 2 election: Requested vote from peers 391a359744cb4913b6a7a3fdfaff318a (127.31.93.62:45495), e0326546ecbb435d8d754a858e366376 (127.31.93.60:35871)
I20260809 04:50:37.666826 10710 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e0326546ecbb435d8d754a858e366376 in current term 2: Already voted for candidate 025ce77beaad4885a83c296e1694006e in this term.
I20260809 04:50:37.667070 11022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "025ce77beaad4885a83c296e1694006e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e0326546ecbb435d8d754a858e366376"
I20260809 04:50:37.667339 11022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 1 FOLLOWER]: Advancing to term 2
I20260809 04:50:37.667694 10981 leader_election.cc:302] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0326546ecbb435d8d754a858e366376; no voters: 025ce77beaad4885a83c296e1694006e, 391a359744cb4913b6a7a3fdfaff318a
I20260809 04:50:37.667802 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.667274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.0.0.1:43138)
0809 04:50:37.667574 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.667666 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260809 04:50:37.668792 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.668409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.0.0.1:43142)
0809 04:50:37.668588 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.668668 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 04:50:37.668954 10654 leader_election.cc:334] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 391a359744cb4913b6a7a3fdfaff318a (127.31.93.62:45495): Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
I20260809 04:50:37.669945 11022 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025ce77beaad4885a83c296e1694006e in term 2.
I20260809 04:50:37.670111 11847 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:50:37.670337 10655 leader_election.cc:302] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 025ce77beaad4885a83c296e1694006e, e0326546ecbb435d8d754a858e366376; no voters: 391a359744cb4913b6a7a3fdfaff318a
I20260809 04:50:37.670616 11850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 2 FOLLOWER]: Leader election won for term 2
I20260809 04:50:37.671036 11850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [term 2 LEADER]: Becoming Leader. State: Replica: 025ce77beaad4885a83c296e1694006e, State: Running, Role: LEADER
I20260809 04:50:37.671380 11850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } }
I20260809 04:50:37.673236 11867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 025ce77beaad4885a83c296e1694006e. Latest consensus state: current_term: 2 leader_uuid: "025ce77beaad4885a83c296e1694006e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } } }
I20260809 04:50:37.673647 11867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:37.674255 11871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:50:37.678534 11871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:50:37.679381 11871 catalog_manager.cc:1259] Loaded cluster ID: bbc24b55091f4398b1e1f66b7eb68697
I20260809 04:50:37.679455 11871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:50:37.680593 11871 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:50:37.681421 11871 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: Loaded TSK: 0
I20260809 04:50:37.682255 11871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:50:37.736966 10687 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:50:37.743808 10687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:50:37.747812 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.747363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.0.0.1:43152)
0809 04:50:37.747535 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.747636 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 04:50:37.747943 11022 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [term 2 FOLLOWER]: Refusing update from remote peer 025ce77beaad4885a83c296e1694006e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260809 04:50:37.748624 11867 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 04:50:37.748760 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.748410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.0.0.1:43156)
0809 04:50:37.748540 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.748639 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 04:50:37.748953 10654 consensus_peers.cc:487] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e -> Peer 391a359744cb4913b6a7a3fdfaff318a (127.31.93.62:45495): Couldn't send request to peer 391a359744cb4913b6a7a3fdfaff318a. Status: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 04:50:37.751096 11847 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 025ce77beaad4885a83c296e1694006e. Latest consensus state: current_term: 2 leader_uuid: "025ce77beaad4885a83c296e1694006e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } } }
I20260809 04:50:37.751335 11847 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:50:37.752826 11867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "025ce77beaad4885a83c296e1694006e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } } }
I20260809 04:50:37.753109 11874 mvcc.cc:205] Tried to move back new op lower bound from 7316484250609045504 to 7316484250305388544. Current Snapshot: MvccSnapshot[applied={T|T < 7316484250609045504}]
I20260809 04:50:37.753194 11867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:37.753706 11847 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "025ce77beaad4885a83c296e1694006e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391a359744cb4913b6a7a3fdfaff318a" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45495 } } peers { permanent_uuid: "025ce77beaad4885a83c296e1694006e" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 41063 } attrs { promote: false } } peers { permanent_uuid: "e0326546ecbb435d8d754a858e366376" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 35871 } attrs { promote: false } } }
I20260809 04:50:37.753980 11847 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0326546ecbb435d8d754a858e366376 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:50:37.769469 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.760922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:40459 (local address 127.0.0.1:58834)
0809 04:50:37.761205 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.761223 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:37.761358 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:37.761915 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:37.761921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:37.761953 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:37.762162 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:37.762178 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.764232 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.764239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:37.766369 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:37.766387 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.768285 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.768292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:37.768503 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:37.768558 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 04:50:37.768641 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 04:50:37.769141 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260809 04:50:37.770587 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.761078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.31.93.2:40459)
0809 04:50:37.761589 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:37.761595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:37.761613 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:50:37.761685 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:37.761690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:37.761765 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:37.761870 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:37.762336 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.764057 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.766589 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.767921 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.768047 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:37.768170 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:37.770097 (+ 1927us) server_negotiation.cc:299] Negotiation successful
0809 04:50:37.770271 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":271,"threads_started":1}
I20260809 04:50:37.771337 11394 tablet_service.cc:1467] Processing CreateTablet for tablet cba364be6857470a816c8abc93bc0e0d (DEFAULT_TABLE table=test-table [id=a493a05a0cda4977827dfdde706352bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:37.771883 11394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cba364be6857470a816c8abc93bc0e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:37.778252 11887 tablet_bootstrap.cc:492] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be: Bootstrap starting.
I20260809 04:50:37.779728 11887 tablet_bootstrap.cc:654] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:37.780678 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.761667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:43813 (local address 127.0.0.1:59306)
0809 04:50:37.762246 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.762263 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:37.762389 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:37.774881 (+ 12492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:37.774886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:37.774911 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:37.775146 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:37.775155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.776265 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.776270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:37.778146 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:37.778161 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.779892 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.779899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:37.780104 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:37.780143 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 04:50:37.780210 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 04:50:37.780357 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472}
I20260809 04:50:37.781785 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.761780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.31.93.3:43813)
0809 04:50:37.774612 (+ 12832us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:37.774617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:37.774632 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:50:37.774688 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:37.774692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:37.774756 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:37.774842 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:37.775283 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.776084 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.778372 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.779648 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.781172 (+ 1524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:37.781295 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:37.781348 (+ 53us) server_negotiation.cc:299] Negotiation successful
0809 04:50:37.781494 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12737,"thread_start_us":12659,"threads_started":1}
I20260809 04:50:37.782609 11887 tablet_bootstrap.cc:492] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be: No bootstrap required, opened a new log
I20260809 04:50:37.782528 11610 tablet_service.cc:1467] Processing CreateTablet for tablet cba364be6857470a816c8abc93bc0e0d (DEFAULT_TABLE table=test-table [id=a493a05a0cda4977827dfdde706352bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:37.782752 11887 ts_tablet_manager.cc:1397] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 04:50:37.783010 11610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cba364be6857470a816c8abc93bc0e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:37.783401 11887 raft_consensus.cc:348] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } }
I20260809 04:50:37.783595 11887 raft_consensus.cc:374] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:37.783685 11887 raft_consensus.cc:729] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3480182938594d0a84d8c3986d8ee6be, State: Initialized, Role: FOLLOWER
I20260809 04:50:37.783921 11887 consensus_queue.cc:260] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } }
I20260809 04:50:37.784654 11887 ts_tablet_manager.cc:1428] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 04:50:37.787814 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.761998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:33309 (local address 127.0.0.1:43266)
0809 04:50:37.769692 (+ 7694us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.769707 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:37.769802 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:37.782410 (+ 12608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:37.782414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:37.782441 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:37.782613 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:37.782624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.783850 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.783855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:37.785597 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:37.785612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.787111 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.787118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:37.787339 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:37.787373 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 04:50:37.787436 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 04:50:37.787557 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7602,"thread_start_us":8809,"threads_started":1}
I20260809 04:50:37.788597 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.762556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.31.93.1:33309)
0809 04:50:37.782065 (+ 19509us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:37.782071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:37.782090 (+ 19us) server_negotiation.cc:407] Connection header received
0809 04:50:37.782160 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:37.782165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:37.782252 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:37.782384 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:37.782786 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.783694 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.785788 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.786928 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.788024 (+ 1096us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:37.788133 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:37.788180 (+ 47us) server_negotiation.cc:299] Negotiation successful
0809 04:50:37.788307 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19388,"thread_start_us":19279,"threads_started":1}
I20260809 04:50:37.789085 11493 heartbeater.cc:503] Master 127.31.93.61:41063 was elected leader, sending a full tablet report...
I20260809 04:50:37.789314 11893 tablet_bootstrap.cc:492] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee: Bootstrap starting.
I20260809 04:50:37.789908 11168 tablet_service.cc:1467] Processing CreateTablet for tablet cba364be6857470a816c8abc93bc0e0d (DEFAULT_TABLE table=test-table [id=a493a05a0cda4977827dfdde706352bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:37.790372 11168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cba364be6857470a816c8abc93bc0e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:37.790964 11893 tablet_bootstrap.cc:654] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:37.793025 11893 tablet_bootstrap.cc:492] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee: No bootstrap required, opened a new log
I20260809 04:50:37.793176 11893 ts_tablet_manager.cc:1397] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260809 04:50:37.793946 11893 raft_consensus.cc:348] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } }
I20260809 04:50:37.794179 11893 raft_consensus.cc:374] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:37.795372 11893 raft_consensus.cc:729] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ecb8b991eb4fceae5fb19e581391ee, State: Initialized, Role: FOLLOWER
I20260809 04:50:37.795648 11893 consensus_queue.cc:260] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } }
I20260809 04:50:37.800040 11899 tablet_bootstrap.cc:492] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c: Bootstrap starting.
I20260809 04:50:37.802068 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.789579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.2:40133)
0809 04:50:37.801760 (+ 12181us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.801879 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12083,"threads_started":1}
I20260809 04:50:37.802256 11899 tablet_bootstrap.cc:654] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:37.803866 11893 ts_tablet_manager.cc:1428] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee: Time spent starting tablet: real 0.011s user 0.000s sys 0.010s
I20260809 04:50:37.804838 11901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.803915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.3:33573)
0809 04:50:37.804472 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.804588 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":274,"threads_started":1}
I20260809 04:50:37.805023 11899 tablet_bootstrap.cc:492] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c: No bootstrap required, opened a new log
I20260809 04:50:37.805162 11899 ts_tablet_manager.cc:1397] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 04:50:37.805805 11901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.805449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.3:47461)
0809 04:50:37.805600 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.805679 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 04:50:37.805891 11899 raft_consensus.cc:348] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } }
I20260809 04:50:37.806114 11899 raft_consensus.cc:374] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:37.806185 11899 raft_consensus.cc:729] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67bceba2f21444d69fdc7d07a7242c3c, State: Initialized, Role: FOLLOWER
I20260809 04:50:37.806381 11899 consensus_queue.cc:260] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } }
I20260809 04:50:37.807126 11899 ts_tablet_manager.cc:1428] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 04:50:37.808594 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.807452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.1:50873)
0809 04:50:37.808195 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.808308 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":574,"threads_started":1}
I20260809 04:50:37.809415 11285 heartbeater.cc:503] Master 127.31.93.61:41063 was elected leader, sending a full tablet report...
I20260809 04:50:37.809872 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.809490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.1:46479)
0809 04:50:37.809648 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.809740 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 04:50:37.890527 11890 raft_consensus.cc:482] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:50:37.890727 11890 raft_consensus.cc:504] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } }
I20260809 04:50:37.892230 11890 leader_election.cc:288] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ecb8b991eb4fceae5fb19e581391ee (127.31.93.3:43813), 67bceba2f21444d69fdc7d07a7242c3c (127.31.93.1:33309)
I20260809 04:50:37.899971 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.892535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:35349 (local address 127.31.93.3:43813)
0809 04:50:37.892750 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:37.892757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:37.893027 (+ 270us) server_negotiation.cc:407] Connection header received
0809 04:50:37.893231 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:37.893235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:37.893311 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:37.893408 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:37.893982 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.895115 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.897754 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.898963 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.899086 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:37.899214 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:37.899568 (+ 354us) server_negotiation.cc:299] Negotiation successful
0809 04:50:37.899707 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 04:50:37.899978 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.892645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:36033 (local address 127.31.93.1:33309)
0809 04:50:37.892846 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:37.892851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:37.893195 (+ 344us) server_negotiation.cc:407] Connection header received
0809 04:50:37.893328 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:37.893333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:37.893408 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:37.893487 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:37.894049 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.895099 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.897748 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.898956 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.899105 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:37.899214 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:37.899568 (+ 354us) server_negotiation.cc:299] Negotiation successful
0809 04:50:37.899703 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 04:50:37.900111 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.892450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:43813 (local address 127.31.93.2:35349)
0809 04:50:37.892911 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.892934 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:37.893094 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:37.893429 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:37.893432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:37.893456 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:37.893833 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:37.893845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.895227 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.895232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:37.897611 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:37.897627 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.899172 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.899178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:37.899383 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:37.899463 (+ 80us) client_negotiation.cc:769] Sending connection context
0809 04:50:37.899539 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 04:50:37.899752 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":185}
I20260809 04:50:37.900493 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.892481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:33309 (local address 127.31.93.2:36033)
0809 04:50:37.893105 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.893119 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:37.893224 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:37.893509 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:37.893513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:37.893538 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:37.893911 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:37.893921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.895227 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.895232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:37.897616 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:37.897627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.899175 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.899180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:37.899383 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:37.899443 (+ 60us) client_negotiation.cc:769] Sending connection context
0809 04:50:37.899539 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 04:50:37.899749 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":216,"thread_start_us":340,"threads_started":1}
I20260809 04:50:37.900707 11647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba364be6857470a816c8abc93bc0e0d" candidate_uuid: "3480182938594d0a84d8c3986d8ee6be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" is_pre_election: true
I20260809 04:50:37.900978 11647 raft_consensus.cc:2454] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3480182938594d0a84d8c3986d8ee6be in term 0.
I20260809 04:50:37.901491 11315 leader_election.cc:302] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3480182938594d0a84d8c3986d8ee6be, c5ecb8b991eb4fceae5fb19e581391ee; no voters:
I20260809 04:50:37.901430 11204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba364be6857470a816c8abc93bc0e0d" candidate_uuid: "3480182938594d0a84d8c3986d8ee6be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67bceba2f21444d69fdc7d07a7242c3c" is_pre_election: true
I20260809 04:50:37.901865 11204 raft_consensus.cc:2454] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3480182938594d0a84d8c3986d8ee6be in term 0.
I20260809 04:50:37.901902 11890 raft_consensus.cc:2781] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:50:37.902016 11890 raft_consensus.cc:482] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:50:37.902104 11890 raft_consensus.cc:3037] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:37.905381 11890 raft_consensus.cc:504] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } }
I20260809 04:50:37.905939 11890 leader_election.cc:288] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [CANDIDATE]: Term 1 election: Requested vote from peers c5ecb8b991eb4fceae5fb19e581391ee (127.31.93.3:43813), 67bceba2f21444d69fdc7d07a7242c3c (127.31.93.1:33309)
W20260809 04:50:37.906345 11507 tablet.cc:2369] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:50:37.906415 11204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba364be6857470a816c8abc93bc0e0d" candidate_uuid: "3480182938594d0a84d8c3986d8ee6be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67bceba2f21444d69fdc7d07a7242c3c"
I20260809 04:50:37.906425 11647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba364be6857470a816c8abc93bc0e0d" candidate_uuid: "3480182938594d0a84d8c3986d8ee6be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ecb8b991eb4fceae5fb19e581391ee"
I20260809 04:50:37.906618 11204 raft_consensus.cc:3037] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:37.906620 11647 raft_consensus.cc:3037] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:37.908526 11647 raft_consensus.cc:2454] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3480182938594d0a84d8c3986d8ee6be in term 1.
I20260809 04:50:37.908686 11204 raft_consensus.cc:2454] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3480182938594d0a84d8c3986d8ee6be in term 1.
I20260809 04:50:37.908878 11315 leader_election.cc:302] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3480182938594d0a84d8c3986d8ee6be, c5ecb8b991eb4fceae5fb19e581391ee; no voters:
I20260809 04:50:37.909128 11890 raft_consensus.cc:2781] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:37.909524 11890 raft_consensus.cc:686] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [term 1 LEADER]: Becoming Leader. State: Replica: 3480182938594d0a84d8c3986d8ee6be, State: Running, Role: LEADER
I20260809 04:50:37.910066 11890 consensus_queue.cc:237] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } } peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } }
I20260809 04:50:37.910516 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.910067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.2:52541)
0809 04:50:37.910231 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.910361 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 04:50:37.913838 10687 catalog_manager.cc:5495] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be reported cstate change: term changed from 0 to 1, leader changed from <none> to 3480182938594d0a84d8c3986d8ee6be (127.31.93.2). New cstate: current_term: 1 leader_uuid: "3480182938594d0a84d8c3986d8ee6be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3480182938594d0a84d8c3986d8ee6be" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20260809 04:50:37.964968 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.960413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.31.93.2:40459)
0809 04:50:37.960607 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:37.960611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:37.960628 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:50:37.960711 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:37.960714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:37.960790 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:37.960910 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:50:37.961184 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.961867 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.963075 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:37.963574 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:37.963667 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:37.964026 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:50:37.964557 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:50:37.964661 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:37.964697 (+ 36us) server_negotiation.cc:299] Negotiation successful
0809 04:50:37.964757 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:50:37.968988 11647 raft_consensus.cc:1261] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee [term 1 FOLLOWER]: Refusing update from remote peer 3480182938594d0a84d8c3986d8ee6be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:37.969255 11204 raft_consensus.cc:1261] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c [term 1 FOLLOWER]: Refusing update from remote peer 3480182938594d0a84d8c3986d8ee6be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:37.969696 11908 consensus_queue.cc:1035] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ecb8b991eb4fceae5fb19e581391ee" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 43813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:37.970153 11890 consensus_queue.cc:1035] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [LEADER]: Connected to new peer: Peer: permanent_uuid: "67bceba2f21444d69fdc7d07a7242c3c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:37.971638 11730 heartbeater.cc:503] Master 127.31.93.61:41063 was elected leader, sending a full tablet report...
I20260809 04:50:37.972151 11901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.971723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.3:38673)
0809 04:50:37.971895 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.971987 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 04:50:37.973644 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.973064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.1:35903)
0809 04:50:37.973251 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.973381 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 04:50:37.978545 11901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.978053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.3:46881)
0809 04:50:37.978276 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.978381 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260809 04:50:37.982837 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.982451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.2:45151)
0809 04:50:37.982604 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.982693 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260809 04:50:37.991698 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.991295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.2:35357)
0809 04:50:37.991437 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.991522 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 04:50:37.994576 10687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:37.994869 10687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:37.995023 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:37.994575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.31.93.1:50735)
0809 04:50:37.994784 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:37.994868 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260809 04:50:38.001348 10687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 025ce77beaad4885a83c296e1694006e: Sending DeleteTablet for 3 replicas of tablet cba364be6857470a816c8abc93bc0e0d
I20260809 04:50:38.002668 11168 tablet_service.cc:1514] Processing DeleteTablet for tablet cba364be6857470a816c8abc93bc0e0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:37 UTC) from {username='oryx', principal='oryx/127.31.93.61@KRBTEST.COM'} at 127.0.0.1:43266
I20260809 04:50:38.002781 11394 tablet_service.cc:1514] Processing DeleteTablet for tablet cba364be6857470a816c8abc93bc0e0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:37 UTC) from {username='oryx', principal='oryx/127.31.93.61@KRBTEST.COM'} at 127.0.0.1:58834
I20260809 04:50:38.003264 11166 tablet_service.cc:1514] Processing DeleteTablet for tablet cba364be6857470a816c8abc93bc0e0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:37 UTC) from {username='oryx', principal='oryx/127.31.93.61@KRBTEST.COM'} at 127.0.0.1:43266
I20260809 04:50:38.003777 11924 tablet_replica.cc:330] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be: stopping tablet replica
I20260809 04:50:38.004104 11924 raft_consensus.cc:2229] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:50:38.004110 11610 tablet_service.cc:1514] Processing DeleteTablet for tablet cba364be6857470a816c8abc93bc0e0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:37 UTC) from {username='oryx', principal='oryx/127.31.93.61@KRBTEST.COM'} at 127.0.0.1:59306
I20260809 04:50:38.005018 11924 raft_consensus.cc:2258] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:38.005854 11926 tablet_replica.cc:330] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee: stopping tablet replica
I20260809 04:50:38.006102 11926 raft_consensus.cc:2229] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:38.006464 11926 raft_consensus.cc:2258] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:38.007009 11924 ts_tablet_manager.cc:1905] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:38.007454 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 10961
I20260809 04:50:38.013628 11924 ts_tablet_manager.cc:1918] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:38.013777 11924 log.cc:1196] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/cba364be6857470a816c8abc93bc0e0d
I20260809 04:50:38.007802 11926 ts_tablet_manager.cc:1905] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:38.014251 11924 ts_tablet_manager.cc:1939] T cba364be6857470a816c8abc93bc0e0d P 3480182938594d0a84d8c3986d8ee6be: Deleting consensus metadata
I20260809 04:50:38.015354 10654 catalog_manager.cc:4841] TS 3480182938594d0a84d8c3986d8ee6be (127.31.93.2:40459): tablet cba364be6857470a816c8abc93bc0e0d (table test-table [id=a493a05a0cda4977827dfdde706352bb]) successfully deleted
I20260809 04:50:38.018440 11923 tablet_replica.cc:330] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c: stopping tablet replica
I20260809 04:50:38.018635 11923 raft_consensus.cc:2229] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:38.018929 11923 raft_consensus.cc:2258] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:38.020025 11923 ts_tablet_manager.cc:1905] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:38.023787 11926 ts_tablet_manager.cc:1918] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:38.024142 11926 log.cc:1196] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/cba364be6857470a816c8abc93bc0e0d
I20260809 04:50:38.024597 11926 ts_tablet_manager.cc:1939] T cba364be6857470a816c8abc93bc0e0d P c5ecb8b991eb4fceae5fb19e581391ee: Deleting consensus metadata
I20260809 04:50:38.025684 10654 catalog_manager.cc:4841] TS c5ecb8b991eb4fceae5fb19e581391ee (127.31.93.3:43813): tablet cba364be6857470a816c8abc93bc0e0d (table test-table [id=a493a05a0cda4977827dfdde706352bb]) successfully deleted
I20260809 04:50:38.026772 11923 ts_tablet_manager.cc:1918] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:38.026916 11923 log.cc:1196] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/cba364be6857470a816c8abc93bc0e0d
I20260809 04:50:38.027393 11923 ts_tablet_manager.cc:1939] T cba364be6857470a816c8abc93bc0e0d P 67bceba2f21444d69fdc7d07a7242c3c: Deleting consensus metadata
I20260809 04:50:38.028407 10658 catalog_manager.cc:4841] TS 67bceba2f21444d69fdc7d07a7242c3c (127.31.93.1:33309): tablet cba364be6857470a816c8abc93bc0e0d (table test-table [id=a493a05a0cda4977827dfdde706352bb]) successfully deleted
W20260809 04:50:38.028785 10658 catalog_manager.cc:4808] TS 67bceba2f21444d69fdc7d07a7242c3c (127.31.93.1:33309): delete failed for tablet cba364be6857470a816c8abc93bc0e0d because the tablet was not found. No further retry: Not found: Tablet not found: cba364be6857470a816c8abc93bc0e0d
I20260809 04:50:38.039422 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:38.008231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.31.93.1:33309)
0809 04:50:38.008393 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:38.008399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:38.024006 (+ 15607us) server_negotiation.cc:407] Connection header received
0809 04:50:38.024284 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:38.024289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:38.024369 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:38.024445 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:38.026346 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:38.027489 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:38.029024 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:38.029631 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:38.029759 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:38.032923 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:38.032939 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:38.032959 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:38.033004 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:38.036753 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:38.037388 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:38.037397 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:38.037401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:38.037557 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:38.037872 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:38.037877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:38.037880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:38.038385 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:38.038681 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:38.038860 (+ 179us) server_negotiation.cc:299] Negotiation successful
0809 04:50:38.039073 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:50:38.040120 11269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:38.184724 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 10961
I20260809 04:50:38.211668 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 11295
I20260809 04:50:38.227043 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:38.212506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.31.93.2:40459)
0809 04:50:38.212693 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:38.212697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:38.212760 (+ 63us) server_negotiation.cc:407] Connection header received
0809 04:50:38.212958 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:38.212962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:38.213030 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:38.213122 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:38.214865 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:38.215652 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:38.217382 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:38.217877 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:38.217976 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:38.221126 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:38.221140 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:38.221152 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:38.221188 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:38.224150 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:38.224929 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:38.224937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:38.224939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:38.225144 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:38.225573 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:38.225581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:38.225583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:38.225941 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:38.226195 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:38.226539 (+ 344us) server_negotiation.cc:299] Negotiation successful
0809 04:50:38.226705 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 04:50:38.227842 11484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:38.319633 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:38.318924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.0.0.1:44190)
0809 04:50:38.319339 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:38.319435 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":211,"threads_started":1}
I20260809 04:50:38.320889 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:38.320562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45495 (local address 127.0.0.1:44192)
0809 04:50:38.320702 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:38.320775 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 04:50:38.349151 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 11295
I20260809 04:50:38.370882 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 11515
I20260809 04:50:38.383123 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:38.371594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.31.93.3:43813)
0809 04:50:38.371789 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:38.371794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:38.371813 (+ 19us) server_negotiation.cc:407] Connection header received
0809 04:50:38.371977 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:38.371981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:38.372056 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:38.372137 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:38.373357 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:38.374083 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:38.375258 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:38.375656 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:38.375746 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:38.377947 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:38.377961 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:38.377977 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:38.378030 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:38.380797 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:38.381363 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:38.381370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:38.381373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:38.381503 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:38.381788 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:38.381791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:38.381792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:38.382141 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:38.382394 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:38.382642 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 04:50:38.382796 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 04:50:38.383754 11719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:38.473814 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 11515
I20260809 04:50:38.496596 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 10460
I20260809 04:50:38.497784 10742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:38.616480 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 10460
I20260809 04:50:38.642336 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 10627
I20260809 04:50:38.643386 11049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:38.757790 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 10627
2026-08-09T04:50:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9885 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:50:38 dist-test-slave-4tdj krb5kdc[11939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:50:38 dist-test-slave-4tdj krb5kdc[11939](info): set up 2 sockets
Aug 09 04:50:38 dist-test-slave-4tdj krb5kdc[11939](info): commencing operation
krb5kdc: starting...
W20260809 04:50:39.968515 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:50:40 dist-test-slave-4tdj krb5kdc[11939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:50:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.62.keytab.
Entry for principal oryx/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.62.keytab.
I20260809 04:50:40.175329 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:41783
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.62.keytab
--principal=oryx/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:41783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:40.491765 11961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:40.492017 11961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:40.492111 11961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:40.498962 11961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:40.499053 11961 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 04:50:40.499097 11961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:40.499162 11961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:40.499212 11961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:40.499271 11961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:50:40.508509 11961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:41783
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.62.keytab
--principal=oryx/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:41783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:40.509596 11961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:40.510870 11961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:40.521734 11972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:40.521703 11969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:40.521997 11968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:40.522699 11961 server_base.cc:1034] running on GCE node
I20260809 04:50:40.523423 11961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:40.524657 11961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:40.525868 11961 hybrid_clock.cc:648] HybridClock initialized: now 1786251040525811 us; error 39 us; skew 500 ppm
I20260809 04:50:40.526165 11961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:40 dist-test-slave-4tdj krb5kdc[11939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251040, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:40.528998 11961 init.cc:377] Logged in from keytab as oryx/127.31.93.62@KRBTEST.COM (short username oryx)
I20260809 04:50:40.530262 11961 webserver.cc:460] Webserver started at http://127.31.93.62:34041/ using document root <none> and password file <none>
I20260809 04:50:40.530925 11961 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:40.531009 11961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:40.531268 11961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:40.534051 11961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "095698cca9684c868920b3920294f85c"
format_stamp: "Formatted at 2026-08-09 04:50:40 on dist-test-slave-4tdj"
server_key: "56a478fba3ddbadae7f250acadd6d606"
server_key_iv: "b042ab353fdf7f9333ab110b991e97bc"
server_key_version: "encryptionkey@0"
I20260809 04:50:40.534823 11961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "095698cca9684c868920b3920294f85c"
format_stamp: "Formatted at 2026-08-09 04:50:40 on dist-test-slave-4tdj"
server_key: "56a478fba3ddbadae7f250acadd6d606"
server_key_iv: "b042ab353fdf7f9333ab110b991e97bc"
server_key_version: "encryptionkey@0"
I20260809 04:50:40.539397 11961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 04:50:40.542541 11979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:40.543762 11961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:50:40.543915 11961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "095698cca9684c868920b3920294f85c"
format_stamp: "Formatted at 2026-08-09 04:50:40 on dist-test-slave-4tdj"
server_key: "56a478fba3ddbadae7f250acadd6d606"
server_key_iv: "b042ab353fdf7f9333ab110b991e97bc"
server_key_version: "encryptionkey@0"
I20260809 04:50:40.544099 11961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:40.560360 11961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:40.563896 11961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:40.564121 11961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:40.583418 11961 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:41783
I20260809 04:50:40.583449 12074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:41783 every 8 connection(s)
I20260809 04:50:40.584617 11961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:50:40.588099 12075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:40.593910 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 11961
I20260809 04:50:40.594092 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:50:40.594764 32116 external_mini_cluster.cc:1428] Setting key 7c8e52d189f790f0cdd87a8687fcfc2c
I20260809 04:50:40.598351 12075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c: Bootstrap starting.
I20260809 04:50:40.601990 12075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:40.603312 12075 log.cc:824] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:40.605878 12075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c: No bootstrap required, opened a new log
Aug 09 04:50:40 dist-test-slave-4tdj krb5kdc[11939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM
I20260809 04:50:40.613104 12075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095698cca9684c868920b3920294f85c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41783 } }
I20260809 04:50:40.613426 12075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:40.613528 12075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095698cca9684c868920b3920294f85c, State: Initialized, Role: FOLLOWER
I20260809 04:50:40.614130 12075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095698cca9684c868920b3920294f85c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41783 } }
I20260809 04:50:40.614387 12075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:50:40.614521 12075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:50:40.614708 12075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:40.616567 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:40.596580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.31.93.62:41783)
0809 04:50:40.597405 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:40.597420 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:40.597466 (+ 46us) server_negotiation.cc:407] Connection header received
0809 04:50:40.599870 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:40.599904 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:40.600392 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:40.600796 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:40.601994 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:40.604007 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:40.605248 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:40.605885 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:40.606047 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:40.609696 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:40.609706 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:40.609719 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:40.609757 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:40.612697 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:40.613301 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:40.613307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:40.613313 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:40.613511 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:40.613956 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:40.613960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:40.613962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:40.614538 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:40.614788 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:40.615240 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 04:50:40.615577 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":341,"threads_started":1}
I20260809 04:50:40.617177 12075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095698cca9684c868920b3920294f85c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41783 } }
I20260809 04:50:40.617641 12075 leader_election.cc:302] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 095698cca9684c868920b3920294f85c; no voters:
I20260809 04:50:40.618263 12075 leader_election.cc:288] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:50:40.618367 12081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:40.620216 12075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:40.621968 12081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [term 1 LEADER]: Becoming Leader. State: Replica: 095698cca9684c868920b3920294f85c, State: Running, Role: LEADER
I20260809 04:50:40.622516 12081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095698cca9684c868920b3920294f85c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41783 } }
I20260809 04:50:40.625329 12085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 095698cca9684c868920b3920294f85c. Latest consensus state: current_term: 1 leader_uuid: "095698cca9684c868920b3920294f85c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095698cca9684c868920b3920294f85c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41783 } } }
I20260809 04:50:40.625649 12085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:40.626217 12082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "095698cca9684c868920b3920294f85c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095698cca9684c868920b3920294f85c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41783 } } }
I20260809 04:50:40.626494 12082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c [sys.catalog]: This master's current role is: LEADER
W20260809 04:50:40.630705 12104 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:50:40.630823 12104 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:50:40.630944 12105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:50:40.632349 12105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:50:40.640179 12105 catalog_manager.cc:1347] Generated new cluster ID: 6ae9fa4817794fc3ad4b1f1c4fc629ce
I20260809 04:50:40.640343 12105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:50:40.660821 12105 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:50:40.662029 12105 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:50:40.669291 12105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c: Generated new TSK 0
I20260809 04:50:40.670035 12105 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.1.keytab.
Entry for principal oryx/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.1.keytab.
I20260809 04:50:40.729364 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:41783
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.1.keytab
--principal=oryx/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:41.053194 12116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:41.053443 12116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:41.053591 12116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:41.060683 12116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:41.060781 12116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:41.060907 12116 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:50:41.061019 12116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:50:41.070729 12116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.1.keytab
--principal=oryx/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.93.62:41783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:41.071965 12116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:41.073428 12116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:41.086052 12123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:41.086025 12130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:41.086104 12124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:41.086932 12116 server_base.cc:1034] running on GCE node
I20260809 04:50:41.087484 12116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:41.088346 12116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:41.089545 12116 hybrid_clock.cc:648] HybridClock initialized: now 1786251041089480 us; error 42 us; skew 500 ppm
I20260809 04:50:41.089823 12116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:41 dist-test-slave-4tdj krb5kdc[11939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251041, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:41.092816 12116 init.cc:377] Logged in from keytab as oryx/127.31.93.1@KRBTEST.COM (short username oryx)
I20260809 04:50:41.094120 12116 webserver.cc:460] Webserver started at http://127.31.93.1:37527/ using document root <none> and password file <none>
I20260809 04:50:41.094790 12116 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:41.094892 12116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:41.095194 12116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:41.097944 12116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "43eea014c2f4402db991371ba0ed7c29"
format_stamp: "Formatted at 2026-08-09 04:50:41 on dist-test-slave-4tdj"
server_key: "f72fcb78747584c69643ec8f2aa3aca2"
server_key_iv: "df152fa5839ee4c0908f266adf2abc96"
server_key_version: "encryptionkey@0"
I20260809 04:50:41.098726 12116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "43eea014c2f4402db991371ba0ed7c29"
format_stamp: "Formatted at 2026-08-09 04:50:41 on dist-test-slave-4tdj"
server_key: "f72fcb78747584c69643ec8f2aa3aca2"
server_key_iv: "df152fa5839ee4c0908f266adf2abc96"
server_key_version: "encryptionkey@0"
I20260809 04:50:41.103554 12116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:50:41.106921 12139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:41.108161 12116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:50:41.108304 12116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "43eea014c2f4402db991371ba0ed7c29"
format_stamp: "Formatted at 2026-08-09 04:50:41 on dist-test-slave-4tdj"
server_key: "f72fcb78747584c69643ec8f2aa3aca2"
server_key_iv: "df152fa5839ee4c0908f266adf2abc96"
server_key_version: "encryptionkey@0"
I20260809 04:50:41.108505 12116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:41.124884 12116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:41.129022 12116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:41.129264 12116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:41.131820 12116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:41.131896 12116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:41.132037 12116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:41.132110 12116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:50:41 dist-test-slave-4tdj krb5kdc[11939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251041, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM
I20260809 04:50:41.162621 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:41.132315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:38271 (local address 127.31.93.62:41783)
0809 04:50:41.144613 (+ 12298us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:41.144620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:41.144643 (+ 23us) server_negotiation.cc:407] Connection header received
0809 04:50:41.144729 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:41.144736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:41.144836 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:41.144962 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:41.147329 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:41.148201 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:41.150599 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:41.151123 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:41.151260 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:41.155902 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:41.155908 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:41.155913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:41.155960 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:41.159336 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:41.160247 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:41.160253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:41.160257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:41.160416 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:41.160819 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:41.160824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:41.160828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:41.161188 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:41.161340 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:41.161943 (+ 603us) server_negotiation.cc:299] Negotiation successful
0809 04:50:41.162174 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12152,"thread_start_us":12761,"threads_started":1}
I20260809 04:50:41.163380 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:41.132875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41783 (local address 127.31.93.1:38271)
0809 04:50:41.133712 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:41.133764 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:41.135656 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:41.145312 (+ 9656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:41.145324 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:41.146172 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:41.147087 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:41.147115 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:41.148362 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:41.148366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:41.150461 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:41.150474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:41.151592 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:41.151601 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:41.151887 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:41.152710 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:41.152746 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:41.155633 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:41.159498 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:41.159507 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:41.159526 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:41.160098 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:41.160522 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:41.160525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:41.160528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:41.160722 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:41.161293 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:41.161299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:41.161616 (+ 317us) client_negotiation.cc:769] Sending connection context
0809 04:50:41.161839 (+ 223us) client_negotiation.cc:240] Negotiation successful
0809 04:50:41.162198 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":348,"threads_started":1}
I20260809 04:50:41.174794 12116 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:41733
I20260809 04:50:41.174885 12331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:41733 every 8 connection(s)
I20260809 04:50:41.176270 12116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:50:41.178491 12332 heartbeater.cc:347] Connected to a master server at 127.31.93.62:41783
I20260809 04:50:41.178714 12332 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:41.179383 12332 heartbeater.cc:511] Master 127.31.93.62:41783 requested a full tablet report, sending...
I20260809 04:50:41.180505 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 12116
I20260809 04:50:41.180987 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:50:41.181653 12007 ts_manager.cc:194] Registered new tserver with Master: 43eea014c2f4402db991371ba0ed7c29 (127.31.93.1:41733)
I20260809 04:50:41.181864 32116 external_mini_cluster.cc:1428] Setting key dd05e1525e5faeecbc69c6a500898688
I20260809 04:50:41.183477 12007 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:38271
I20260809 04:50:41.187801 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:41.172274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:40721 (local address 127.31.93.62:41783)
0809 04:50:41.172449 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:41.172455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:41.173332 (+ 877us) server_negotiation.cc:407] Connection header received
0809 04:50:41.173438 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:41.173443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:41.173528 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:41.173631 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:41.175015 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:41.176031 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:41.177671 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:41.178180 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:41.178313 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:41.180071 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:41.180078 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:41.180083 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:41.180118 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:41.182947 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:41.183571 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:41.183577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:41.183581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:41.183753 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:41.186439 (+ 2686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:41.186444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:41.186447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:41.186721 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:41.186861 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:41.187261 (+ 400us) server_negotiation.cc:299] Negotiation successful
0809 04:50:41.187471 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:50:41.187853 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:41.172444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41783 (local address 127.31.93.1:40721)
0809 04:50:41.173224 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:41.173244 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:41.173350 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:41.173682 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:41.173687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:41.174350 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:41.174806 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:41.174821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:41.176158 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:41.176162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:41.177494 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:41.177508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:41.178390 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:41.178397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:41.178566 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:41.179122 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:41.179141 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:41.179871 (+ 730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:41.183064 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:41.183069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:41.183074 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:41.183455 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:41.183877 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:41.183881 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:41.183885 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:41.184038 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:41.186838 (+ 2800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:41.186843 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:41.187006 (+ 163us) client_negotiation.cc:769] Sending connection context
0809 04:50:41.187121 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 04:50:41.187402 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":477,"threads_started":1}
WARNING: no policy specified for oryx/127.31.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.2.keytab.
Entry for principal oryx/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.2.keytab.
I20260809 04:50:41.230616 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:41783
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.2.keytab
--principal=oryx/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:41.544441 12340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:41.544700 12340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:41.544831 12340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:41.551828 12340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:41.551925 12340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:41.552042 12340 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:50:41.552137 12340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:50:41.561326 12340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.2.keytab
--principal=oryx/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.93.62:41783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:41.562513 12340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:41.563805 12340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:41.575744 12348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:41.575809 12353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:41.575775 12349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:41.576687 12340 server_base.cc:1034] running on GCE node
I20260809 04:50:41.577201 12340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:41.577991 12340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:41.579195 12340 hybrid_clock.cc:648] HybridClock initialized: now 1786251041579134 us; error 31 us; skew 500 ppm
I20260809 04:50:41.579463 12340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:41 dist-test-slave-4tdj krb5kdc[11939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251041, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:41.582410 12340 init.cc:377] Logged in from keytab as oryx/127.31.93.2@KRBTEST.COM (short username oryx)
I20260809 04:50:41.584036 12340 webserver.cc:460] Webserver started at http://127.31.93.2:38553/ using document root <none> and password file <none>
I20260809 04:50:41.584853 12340 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:41.584972 12340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:41.585359 12340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:41.589021 12340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "c1d65eb4f9024c81bc2b53669a9e576e"
format_stamp: "Formatted at 2026-08-09 04:50:41 on dist-test-slave-4tdj"
server_key: "a5181a456b29dbe3b25e8d96770cea6f"
server_key_iv: "28faa3e91cab9e79753577d1700c15ce"
server_key_version: "encryptionkey@0"
I20260809 04:50:41.589965 12340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "c1d65eb4f9024c81bc2b53669a9e576e"
format_stamp: "Formatted at 2026-08-09 04:50:41 on dist-test-slave-4tdj"
server_key: "a5181a456b29dbe3b25e8d96770cea6f"
server_key_iv: "28faa3e91cab9e79753577d1700c15ce"
server_key_version: "encryptionkey@0"
I20260809 04:50:41.595913 12340 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260809 04:50:41.600035 12364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:41.601439 12340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 04:50:41.601591 12340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "c1d65eb4f9024c81bc2b53669a9e576e"
format_stamp: "Formatted at 2026-08-09 04:50:41 on dist-test-slave-4tdj"
server_key: "a5181a456b29dbe3b25e8d96770cea6f"
server_key_iv: "28faa3e91cab9e79753577d1700c15ce"
server_key_version: "encryptionkey@0"
I20260809 04:50:41.601807 12340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:41.624020 12340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:41.627514 12340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:41.627736 12340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:41.630220 12340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:41.630317 12340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:41.630471 12340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:41.630555 12340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:50:41 dist-test-slave-4tdj krb5kdc[11939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251041, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM
I20260809 04:50:41.650321 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:41.630575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:34205 (local address 127.31.93.62:41783)
0809 04:50:41.630772 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:41.630780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:41.632191 (+ 1411us) server_negotiation.cc:407] Connection header received
0809 04:50:41.634126 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:41.634133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:41.634235 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:41.634386 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:41.636271 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:41.637394 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:41.639683 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:41.640121 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:41.640235 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:41.643745 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:41.643752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:41.643756 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:41.643796 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:41.646938 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:41.647843 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:41.647847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:41.647849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:41.647978 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:41.648532 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:41.648537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:41.648540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:41.648911 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:41.649042 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:41.649727 (+ 685us) server_negotiation.cc:299] Negotiation successful
0809 04:50:41.649949 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 04:50:41.651458 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:41.631172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41783 (local address 127.31.93.2:34205)
0809 04:50:41.632001 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:41.632046 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:41.633832 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:41.634643 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:41.634655 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:41.635301 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:41.636071 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:41.636095 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:41.637557 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:41.637563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:41.639543 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:41.639555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:41.640363 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:41.640372 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:41.640555 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:41.641095 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:41.641121 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:41.643500 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:41.647107 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:41.647117 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:41.647135 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:41.647686 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:41.648125 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:41.648130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:41.648134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:41.648385 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:41.649035 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:41.649042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:41.649362 (+ 320us) client_negotiation.cc:769] Sending connection context
0809 04:50:41.649597 (+ 235us) client_negotiation.cc:240] Negotiation successful
0809 04:50:41.650042 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":365,"threads_started":1}
I20260809 04:50:41.670276 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:41.656556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41783 (local address 127.31.93.2:48685)
0809 04:50:41.657609 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:41.657628 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:41.657761 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:41.658081 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:41.658086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:41.658764 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:41.659222 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:41.659239 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:41.660353 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:41.660358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:41.661950 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:41.661965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:41.662920 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:41.662928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:41.663053 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:41.663658 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:41.663677 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:41.664570 (+ 893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:41.667646 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:41.667652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:41.667656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:41.668073 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:41.668433 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:41.668438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:41.668442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:41.668563 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:41.669188 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:41.669193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:41.669426 (+ 233us) client_negotiation.cc:769] Sending connection context
0809 04:50:41.669596 (+ 170us) client_negotiation.cc:240] Negotiation successful
0809 04:50:41.669976 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"thread_start_us":752,"threads_started":1}
I20260809 04:50:41.670501 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:41.656560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:48685 (local address 127.31.93.62:41783)
0809 04:50:41.656724 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:41.656729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:41.657712 (+ 983us) server_negotiation.cc:407] Connection header received
0809 04:50:41.657894 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:41.657900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:41.657957 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:41.658034 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:41.659354 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:41.660220 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:41.662086 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:41.662551 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:41.662667 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:41.664738 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:41.664743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:41.664745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:41.664773 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:41.667531 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:41.668198 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:41.668203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:41.668207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:41.668329 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:41.668668 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:41.668673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:41.668677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:41.669088 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:41.669224 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:41.670013 (+ 789us) server_negotiation.cc:299] Negotiation successful
0809 04:50:41.670203 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 04:50:41.676620 12340 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:46389
I20260809 04:50:41.676642 12543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:46389 every 8 connection(s)
I20260809 04:50:41.677772 12340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:50:41.679618 12544 heartbeater.cc:347] Connected to a master server at 127.31.93.62:41783
I20260809 04:50:41.679818 12544 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:41.680316 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 12340
I20260809 04:50:41.680408 12544 heartbeater.cc:511] Master 127.31.93.62:41783 requested a full tablet report, sending...
I20260809 04:50:41.680460 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:50:41.680887 32116 external_mini_cluster.cc:1428] Setting key 8f32306f4103f1c99874a7bc5d26c045
I20260809 04:50:41.681764 12007 ts_manager.cc:194] Registered new tserver with Master: c1d65eb4f9024c81bc2b53669a9e576e (127.31.93.2:46389)
I20260809 04:50:41.682832 12007 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:34205
WARNING: no policy specified for oryx/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.3.keytab.
Entry for principal oryx/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.3.keytab.
I20260809 04:50:41.729005 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:41783
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.3.keytab
--principal=oryx/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:46087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:42.043195 12548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:42.043452 12548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:42.043553 12548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:42.050616 12548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:42.050716 12548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:42.050803 12548 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 04:50:42.050880 12548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:50:42.060220 12548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:46087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.3.keytab
--principal=oryx/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.93.62:41783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:42.061378 12548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:42.062721 12548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:42.075008 12555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:42.075003 12556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:42.075026 12559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:42.075723 12548 server_base.cc:1034] running on GCE node
I20260809 04:50:42.076316 12548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:42.077162 12548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:42.078392 12548 hybrid_clock.cc:648] HybridClock initialized: now 1786251042078330 us; error 43 us; skew 500 ppm
I20260809 04:50:42.078670 12548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:42 dist-test-slave-4tdj krb5kdc[11939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251042, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:42.081555 12548 init.cc:377] Logged in from keytab as oryx/127.31.93.3@KRBTEST.COM (short username oryx)
I20260809 04:50:42.083024 12548 webserver.cc:460] Webserver started at http://127.31.93.3:40133/ using document root <none> and password file <none>
I20260809 04:50:42.083668 12548 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:42.083770 12548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:42.084105 12548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:42.086987 12548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "2f6879888e3a4686870acea2e6d60f2b"
format_stamp: "Formatted at 2026-08-09 04:50:42 on dist-test-slave-4tdj"
server_key: "4a8e7cac8b30345bcdddc5d823b3c40c"
server_key_iv: "43656d6d7402d7ec70337604ef0ce2e6"
server_key_version: "encryptionkey@0"
I20260809 04:50:42.087749 12548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "2f6879888e3a4686870acea2e6d60f2b"
format_stamp: "Formatted at 2026-08-09 04:50:42 on dist-test-slave-4tdj"
server_key: "4a8e7cac8b30345bcdddc5d823b3c40c"
server_key_iv: "43656d6d7402d7ec70337604ef0ce2e6"
server_key_version: "encryptionkey@0"
I20260809 04:50:42.092504 12548 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 04:50:42.095758 12568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:42.096961 12548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 04:50:42.097095 12548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "2f6879888e3a4686870acea2e6d60f2b"
format_stamp: "Formatted at 2026-08-09 04:50:42 on dist-test-slave-4tdj"
server_key: "4a8e7cac8b30345bcdddc5d823b3c40c"
server_key_iv: "43656d6d7402d7ec70337604ef0ce2e6"
server_key_version: "encryptionkey@0"
I20260809 04:50:42.097295 12548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:42.109989 12548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:42.113634 12548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:42.113845 12548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:42.116438 12548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:42.116540 12548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:42.116699 12548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:42.116839 12548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 09 04:50:42 dist-test-slave-4tdj krb5kdc[11939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251042, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM
I20260809 04:50:42.138172 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.116782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:53935 (local address 127.31.93.62:41783)
0809 04:50:42.116971 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:42.116978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:42.118504 (+ 1526us) server_negotiation.cc:407] Connection header received
0809 04:50:42.120778 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:42.120786 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:42.120884 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:42.121010 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:42.123187 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.124255 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.126722 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.127367 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.127508 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.132113 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:42.132118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:42.132122 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:42.132155 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:42.134843 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:42.135589 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:42.135594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:42.135598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:42.135741 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:42.136226 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:42.136232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:42.136236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:42.136547 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:42.136686 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:42.137561 (+ 875us) server_negotiation.cc:299] Negotiation successful
0809 04:50:42.137802 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 04:50:42.139768 12581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.117430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41783 (local address 127.31.93.3:53935)
0809 04:50:42.118329 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:42.118388 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:42.120484 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:42.121282 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:42.121294 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:42.122090 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:42.122996 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.123029 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.124479 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.124485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.126586 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.126602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.127610 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.127622 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.127855 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.128640 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:42.128674 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:42.131868 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:42.134992 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:42.134998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:42.135010 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:42.135452 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:42.135849 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:42.135852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:42.135858 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:42.136036 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:42.136689 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:42.136701 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:42.137089 (+ 388us) client_negotiation.cc:769] Sending connection context
0809 04:50:42.137435 (+ 346us) client_negotiation.cc:240] Negotiation successful
0809 04:50:42.138094 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":398,"threads_started":1}
I20260809 04:50:42.159430 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.145539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:37013 (local address 127.31.93.62:41783)
0809 04:50:42.145687 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:42.145691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:42.146265 (+ 574us) server_negotiation.cc:407] Connection header received
0809 04:50:42.146455 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:42.146461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:42.146530 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:42.146642 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:42.147874 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.148787 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.150761 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.151266 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.151375 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.153576 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:42.153582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:42.153586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:42.153616 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:42.156832 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:42.157436 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:42.157441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:42.157444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:42.157608 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:42.157985 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:42.157990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:42.157994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:42.158318 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:42.158458 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:42.158867 (+ 409us) server_negotiation.cc:299] Negotiation successful
0809 04:50:42.159070 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 04:50:42.159627 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.145579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41783 (local address 127.31.93.3:37013)
0809 04:50:42.146167 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:42.146182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:42.146338 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:42.146704 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:42.146710 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:42.147323 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:42.147739 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.147752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.148931 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.148936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.150559 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.150576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.151443 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.151449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.151613 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.152192 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:42.152209 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:42.153336 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:42.156945 (+ 3609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:42.156953 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:42.156958 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:42.157328 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:42.157742 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:42.157746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:42.157750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:42.157876 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:42.158423 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:42.158428 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:42.158615 (+ 187us) client_negotiation.cc:769] Sending connection context
0809 04:50:42.158747 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 04:50:42.159055 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":319,"threads_started":1}
I20260809 04:50:42.164466 12548 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:39031
I20260809 04:50:42.164505 12769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:39031 every 8 connection(s)
I20260809 04:50:42.165658 12548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:50:42.167544 12771 heartbeater.cc:347] Connected to a master server at 127.31.93.62:41783
I20260809 04:50:42.167739 12771 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:42.168342 12771 heartbeater.cc:511] Master 127.31.93.62:41783 requested a full tablet report, sending...
I20260809 04:50:42.168862 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 12548
I20260809 04:50:42.169040 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:50:42.169482 32116 external_mini_cluster.cc:1428] Setting key 60a45686a11a1e71e7f7eff20999ee26
I20260809 04:50:42.169785 12007 ts_manager.cc:194] Registered new tserver with Master: 2f6879888e3a4686870acea2e6d60f2b (127.31.93.3:39031)
I20260809 04:50:42.170725 12007 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:53935
I20260809 04:50:42.172468 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:50:42.186275 12332 heartbeater.cc:503] Master 127.31.93.62:41783 was elected leader, sending a full tablet report...
Aug 09 04:50:42 dist-test-slave-4tdj krb5kdc[11939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 04:50:42 dist-test-slave-4tdj krb5kdc[11939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:42 dist-test-slave-4tdj krb5kdc[11939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 04:50:42.224198 12150 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:42.214503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.31.93.62:41783)
0809 04:50:42.214727 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:42.214734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:42.214955 (+ 221us) server_negotiation.cc:407] Connection header received
0809 04:50:42.215119 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:42.215123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:42.215200 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:42.215279 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:42.216480 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.217209 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.218399 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.218805 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.218895 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.223927 (+ 5032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54384: BlockingRecv error: recv got EOF from 127.0.0.1:54384 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 09 04:50:42 dist-test-slave-4tdj krb5kdc[11939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM
I20260809 04:50:42.238518 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.226945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.31.93.62:41783)
0809 04:50:42.227135 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:42.227140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:42.227353 (+ 213us) server_negotiation.cc:407] Connection header received
0809 04:50:42.227466 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:42.227469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:42.227535 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:42.227612 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:42.228588 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.229277 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.230452 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.230855 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.230945 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.234127 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:42.234132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:42.234135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:42.234164 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:42.236528 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:42.237049 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:42.237053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:42.237055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:42.237163 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:42.237412 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:42.237415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:42.237417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:42.237692 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:42.237893 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:42.238055 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 04:50:42.238209 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 04:50:42.242100 12007 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:50:42.248858 12007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:50:42.279362 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.260054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:46389 (local address 127.0.0.1:36364)
0809 04:50:42.271451 (+ 11397us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:42.271482 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:42.271600 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:42.272690 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:42.272694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:42.272717 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:42.273161 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.273175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.274906 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.274912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.276808 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.276821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.278441 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.278448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.278625 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.278680 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 04:50:42.278798 (+ 118us) client_negotiation.cc:240] Negotiation successful
0809 04:50:42.278980 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11260,"mutex_wait_us":2,"thread_start_us":11290,"threads_started":1}
I20260809 04:50:42.279801 12805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.259872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:41733 (local address 127.0.0.1:59258)
0809 04:50:42.270108 (+ 10236us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:42.270162 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:42.271558 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:42.272249 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:42.272256 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:42.272285 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:42.272708 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.272722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.274393 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.274398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.277125 (+ 2727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.277143 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.278927 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.278934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.279113 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.279145 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 04:50:42.279248 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 04:50:42.279388 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9976,"mutex_wait_us":52,"spinlock_wait_cycles":19200,"thread_start_us":11214,"threads_started":1}
I20260809 04:50:42.280184 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.260127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59258 (local address 127.31.93.1:41733)
0809 04:50:42.271057 (+ 10930us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:42.271064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:42.271097 (+ 33us) server_negotiation.cc:407] Connection header received
0809 04:50:42.271806 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:42.271815 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:42.272055 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:42.272223 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:42.272931 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.274177 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.277288 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.278721 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.278839 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.279032 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:42.279301 (+ 269us) server_negotiation.cc:299] Negotiation successful
0809 04:50:42.279718 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10783,"thread_start_us":10660,"threads_started":1}
I20260809 04:50:42.280318 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.260503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:39031 (local address 127.0.0.1:60424)
0809 04:50:42.271578 (+ 11075us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:42.271599 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:42.272381 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:42.272922 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:42.272926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:42.272956 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:42.273394 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.273407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.275386 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.275392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.277442 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.277451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.279369 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.279375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.279534 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.279566 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 04:50:42.279667 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 04:50:42.279785 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10971,"mutex_wait_us":11,"thread_start_us":9230,"threads_started":1}
I20260809 04:50:42.280370 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.261177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.31.93.3:39031)
0809 04:50:42.270824 (+ 9647us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:42.270833 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:42.271791 (+ 958us) server_negotiation.cc:407] Connection header received
0809 04:50:42.272539 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:42.272545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:42.272738 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:42.272927 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:42.273612 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.275242 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.277642 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.278873 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.278992 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.279215 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:42.279904 (+ 689us) server_negotiation.cc:299] Negotiation successful
0809 04:50:42.280068 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9471,"thread_start_us":9271,"threads_started":1}
I20260809 04:50:42.280396 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.260591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36364 (local address 127.31.93.2:46389)
0809 04:50:42.272121 (+ 11530us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:42.272128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:42.272152 (+ 24us) server_negotiation.cc:407] Connection header received
0809 04:50:42.272237 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:42.272243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:42.272478 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:42.272688 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:42.273329 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.274739 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.276960 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.278189 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.279703 (+ 1514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.279891 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:42.279967 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 04:50:42.280117 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11383,"thread_start_us":11242,"threads_started":1}
I20260809 04:50:42.283749 12433 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7c7a7f28df4ca1b618f59cc2a7345f (DEFAULT_TABLE table=test-table [id=bd3024be63c74232908684e40c99db6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:42.283924 12655 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7c7a7f28df4ca1b618f59cc2a7345f (DEFAULT_TABLE table=test-table [id=bd3024be63c74232908684e40c99db6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:42.284282 12218 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7c7a7f28df4ca1b618f59cc2a7345f (DEFAULT_TABLE table=test-table [id=bd3024be63c74232908684e40c99db6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:42.284945 12433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7c7a7f28df4ca1b618f59cc2a7345f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:42.285394 12655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7c7a7f28df4ca1b618f59cc2a7345f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:42.285550 12218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7c7a7f28df4ca1b618f59cc2a7345f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:42.295601 12811 tablet_bootstrap.cc:492] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e: Bootstrap starting.
I20260809 04:50:42.298352 12811 tablet_bootstrap.cc:654] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:42.299178 12812 tablet_bootstrap.cc:492] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b: Bootstrap starting.
I20260809 04:50:42.300478 12816 tablet_bootstrap.cc:492] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29: Bootstrap starting.
I20260809 04:50:42.300638 12811 log.cc:824] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:42.302842 12812 tablet_bootstrap.cc:654] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:42.303478 12816 tablet_bootstrap.cc:654] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:42.303967 12812 log.cc:824] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:42.304435 12816 log.cc:824] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:42.304962 12811 tablet_bootstrap.cc:492] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e: No bootstrap required, opened a new log
I20260809 04:50:42.305253 12811 ts_tablet_manager.cc:1397] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 04:50:42.307075 12812 tablet_bootstrap.cc:492] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b: No bootstrap required, opened a new log
I20260809 04:50:42.307365 12812 ts_tablet_manager.cc:1397] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 04:50:42.307401 12816 tablet_bootstrap.cc:492] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29: No bootstrap required, opened a new log
I20260809 04:50:42.307772 12816 ts_tablet_manager.cc:1397] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260809 04:50:42.312520 12811 raft_consensus.cc:348] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.312834 12811 raft_consensus.cc:374] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:42.312942 12811 raft_consensus.cc:729] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d65eb4f9024c81bc2b53669a9e576e, State: Initialized, Role: FOLLOWER
I20260809 04:50:42.313592 12811 consensus_queue.cc:260] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.314551 12544 heartbeater.cc:503] Master 127.31.93.62:41783 was elected leader, sending a full tablet report...
I20260809 04:50:42.316251 12811 ts_tablet_manager.cc:1428] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260809 04:50:42.317333 12812 raft_consensus.cc:348] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.317745 12812 raft_consensus.cc:374] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:42.317579 12816 raft_consensus.cc:348] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.317854 12812 raft_consensus.cc:729] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6879888e3a4686870acea2e6d60f2b, State: Initialized, Role: FOLLOWER
I20260809 04:50:42.317981 12816 raft_consensus.cc:374] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:42.318068 12816 raft_consensus.cc:729] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43eea014c2f4402db991371ba0ed7c29, State: Initialized, Role: FOLLOWER
I20260809 04:50:42.318457 12812 consensus_queue.cc:260] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.318720 12816 consensus_queue.cc:260] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.320623 12816 ts_tablet_manager.cc:1428] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260809 04:50:42.321516 12771 heartbeater.cc:503] Master 127.31.93.62:41783 was elected leader, sending a full tablet report...
I20260809 04:50:42.322098 12812 ts_tablet_manager.cc:1428] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260809 04:50:42.331671 12825 raft_consensus.cc:482] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:50:42.331902 12825 raft_consensus.cc:504] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.333279 12825 leader_election.cc:288] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43eea014c2f4402db991371ba0ed7c29 (127.31.93.1:41733), 2f6879888e3a4686870acea2e6d60f2b (127.31.93.3:39031)
I20260809 04:50:42.341871 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.333628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:40699 (local address 127.31.93.3:39031)
0809 04:50:42.333788 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:42.333794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:42.334591 (+ 797us) server_negotiation.cc:407] Connection header received
0809 04:50:42.334752 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:42.334756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:42.334816 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:42.334895 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:42.335645 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.336711 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.339620 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.340770 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.340901 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.341001 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:42.341531 (+ 530us) server_negotiation.cc:299] Negotiation successful
0809 04:50:42.341644 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 04:50:42.341884 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.333606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:43075 (local address 127.31.93.1:41733)
0809 04:50:42.333789 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:42.333795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:42.334539 (+ 744us) server_negotiation.cc:407] Connection header received
0809 04:50:42.334715 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:42.334721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:42.334808 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:42.334895 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:42.335645 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.336709 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.339628 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.340795 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.340909 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.340999 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:42.341489 (+ 490us) server_negotiation.cc:299] Negotiation successful
0809 04:50:42.341628 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 04:50:42.342028 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.333536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:41733 (local address 127.31.93.2:43075)
0809 04:50:42.334439 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:42.334458 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:42.334595 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:42.334924 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:42.334929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:42.334968 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:42.335509 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.335520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.336826 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.336830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.339480 (+ 2650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.339496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.341021 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.341026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.341304 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.341368 (+ 64us) client_negotiation.cc:769] Sending connection context
0809 04:50:42.341462 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 04:50:42.341587 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":249,"spinlock_wait_cycles":3456,"thread_start_us":431,"threads_started":1}
I20260809 04:50:42.342378 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.333656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:39031 (local address 127.31.93.2:40699)
0809 04:50:42.334439 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:42.334458 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:42.334633 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:42.334924 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:42.334929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:42.334961 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:42.335470 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.335486 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.336823 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.336828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.339488 (+ 2660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.339497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.341013 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.341020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.341308 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.341373 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 04:50:42.341504 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 04:50:42.341602 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":279,"spinlock_wait_cycles":30848,"thread_start_us":383,"threads_started":1}
I20260809 04:50:42.343055 12246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7c7a7f28df4ca1b618f59cc2a7345f" candidate_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eea014c2f4402db991371ba0ed7c29" is_pre_election: true
I20260809 04:50:42.343204 12688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7c7a7f28df4ca1b618f59cc2a7345f" candidate_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6879888e3a4686870acea2e6d60f2b" is_pre_election: true
I20260809 04:50:42.343572 12246 raft_consensus.cc:2454] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d65eb4f9024c81bc2b53669a9e576e in term 0.
I20260809 04:50:42.343639 12688 raft_consensus.cc:2454] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d65eb4f9024c81bc2b53669a9e576e in term 0.
I20260809 04:50:42.344172 12365 leader_election.cc:302] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43eea014c2f4402db991371ba0ed7c29, c1d65eb4f9024c81bc2b53669a9e576e; no voters:
I20260809 04:50:42.344511 12825 raft_consensus.cc:2781] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:50:42.344643 12825 raft_consensus.cc:482] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:50:42.344743 12825 raft_consensus.cc:3037] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:42.346979 12825 raft_consensus.cc:504] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.347675 12825 leader_election.cc:288] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [CANDIDATE]: Term 1 election: Requested vote from peers 43eea014c2f4402db991371ba0ed7c29 (127.31.93.1:41733), 2f6879888e3a4686870acea2e6d60f2b (127.31.93.3:39031)
I20260809 04:50:42.348018 12246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7c7a7f28df4ca1b618f59cc2a7345f" candidate_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eea014c2f4402db991371ba0ed7c29"
I20260809 04:50:42.348095 12688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7c7a7f28df4ca1b618f59cc2a7345f" candidate_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6879888e3a4686870acea2e6d60f2b"
I20260809 04:50:42.348213 12246 raft_consensus.cc:3037] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:42.348271 12688 raft_consensus.cc:3037] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:42.351276 12246 raft_consensus.cc:2454] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d65eb4f9024c81bc2b53669a9e576e in term 1.
I20260809 04:50:42.351276 12688 raft_consensus.cc:2454] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d65eb4f9024c81bc2b53669a9e576e in term 1.
I20260809 04:50:42.351671 12368 leader_election.cc:302] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2f6879888e3a4686870acea2e6d60f2b, 43eea014c2f4402db991371ba0ed7c29, c1d65eb4f9024c81bc2b53669a9e576e; no voters:
I20260809 04:50:42.351984 12825 raft_consensus.cc:2781] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:42.352375 12825 raft_consensus.cc:686] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [term 1 LEADER]: Becoming Leader. State: Replica: c1d65eb4f9024c81bc2b53669a9e576e, State: Running, Role: LEADER
I20260809 04:50:42.352952 12825 consensus_queue.cc:237] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.356974 12007 catalog_manager.cc:5495] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e reported cstate change: term changed from 0 to 1, leader changed from <none> to c1d65eb4f9024c81bc2b53669a9e576e (127.31.93.2). New cstate: current_term: 1 leader_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } health_report { overall_health: UNKNOWN } } }
I20260809 04:50:42.377286 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.372431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.31.93.2:46389)
0809 04:50:42.372609 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:42.372615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:42.372634 (+ 19us) server_negotiation.cc:407] Connection header received
0809 04:50:42.372835 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:42.372838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:42.372911 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:42.373027 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:50:42.373382 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.374036 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.375387 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.375776 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.375877 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.376296 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:50:42.376765 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:50:42.376886 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:42.376989 (+ 103us) server_negotiation.cc:299] Negotiation successful
0809 04:50:42.377059 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:50:42.384346 12688 raft_consensus.cc:1261] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Refusing update from remote peer c1d65eb4f9024c81bc2b53669a9e576e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:42.384346 12246 raft_consensus.cc:1261] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Refusing update from remote peer c1d65eb4f9024c81bc2b53669a9e576e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:42.385310 12825 consensus_queue.cc:1035] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:42.385583 12837 consensus_queue.cc:1035] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:42.396076 12842 mvcc.cc:205] Tried to move back new op lower bound from 7316484269597999104 to 7316484269481512960. Current Snapshot: MvccSnapshot[applied={T|T < 7316484269597999104}]
I20260809 04:50:42.396606 12844 mvcc.cc:205] Tried to move back new op lower bound from 7316484269597999104 to 7316484269481512960. Current Snapshot: MvccSnapshot[applied={T|T < 7316484269597999104}]
I20260809 04:50:42.409991 12007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:42.410204 12007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:42.413630 12007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c: Sending DeleteTablet for 3 replicas of tablet 1b7c7a7f28df4ca1b618f59cc2a7345f
I20260809 04:50:42.414805 12433 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7c7a7f28df4ca1b618f59cc2a7345f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:42 UTC) from {username='oryx', principal='oryx/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:36364
I20260809 04:50:42.414904 12218 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7c7a7f28df4ca1b618f59cc2a7345f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:42 UTC) from {username='oryx', principal='oryx/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:59258
I20260809 04:50:42.414932 12655 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7c7a7f28df4ca1b618f59cc2a7345f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:42 UTC) from {username='oryx', principal='oryx/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:60424
I20260809 04:50:42.417408 12007 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:50:42.418200 12007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:50:42.429816 12432 tablet_service.cc:1467] Processing CreateTablet for tablet ac8f2944c3ae4127a8b4fb76c0686573 (DEFAULT_TABLE table=test-table [id=351a8552c897428d996ddac7f79ec58d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:42.430225 12653 tablet_service.cc:1467] Processing CreateTablet for tablet ac8f2944c3ae4127a8b4fb76c0686573 (DEFAULT_TABLE table=test-table [id=351a8552c897428d996ddac7f79ec58d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:42.430856 12653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8f2944c3ae4127a8b4fb76c0686573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:42.431335 12432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8f2944c3ae4127a8b4fb76c0686573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:42.432346 12855 tablet_replica.cc:330] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e: stopping tablet replica
I20260809 04:50:42.432449 12217 tablet_service.cc:1467] Processing CreateTablet for tablet ac8f2944c3ae4127a8b4fb76c0686573 (DEFAULT_TABLE table=test-table [id=351a8552c897428d996ddac7f79ec58d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:42.432999 12217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac8f2944c3ae4127a8b4fb76c0686573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:42.433226 12858 tablet_replica.cc:330] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29: stopping tablet replica
I20260809 04:50:42.433596 12858 raft_consensus.cc:2229] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:42.434089 12858 raft_consensus.cc:2258] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:42.432804 12855 raft_consensus.cc:2229] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:50:42.435211 12855 raft_consensus.cc:2258] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:42.435596 12856 tablet_replica.cc:330] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b: stopping tablet replica
I20260809 04:50:42.436008 12856 raft_consensus.cc:2229] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:42.437222 12855 ts_tablet_manager.cc:1905] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:42.437726 12856 raft_consensus.cc:2258] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:42.439124 12816 tablet_bootstrap.cc:492] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29: Bootstrap starting.
I20260809 04:50:42.440721 12816 tablet_bootstrap.cc:654] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:42.444026 12856 ts_tablet_manager.cc:1905] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:42.440696 12858 ts_tablet_manager.cc:1905] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:42.445376 12812 tablet_bootstrap.cc:492] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b: Bootstrap starting.
I20260809 04:50:42.446501 12855 ts_tablet_manager.cc:1918] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:42.446568 12811 tablet_bootstrap.cc:492] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e: Bootstrap starting.
I20260809 04:50:42.446656 12855 log.cc:1196] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/1b7c7a7f28df4ca1b618f59cc2a7345f
I20260809 04:50:42.446992 12812 tablet_bootstrap.cc:654] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:42.447173 12855 ts_tablet_manager.cc:1939] T 1b7c7a7f28df4ca1b618f59cc2a7345f P c1d65eb4f9024c81bc2b53669a9e576e: Deleting consensus metadata
I20260809 04:50:42.448237 12811 tablet_bootstrap.cc:654] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:42.449523 11981 catalog_manager.cc:4841] TS c1d65eb4f9024c81bc2b53669a9e576e (127.31.93.2:46389): tablet 1b7c7a7f28df4ca1b618f59cc2a7345f (table test-table [id=bd3024be63c74232908684e40c99db6e]) successfully deleted
I20260809 04:50:42.451812 12858 ts_tablet_manager.cc:1918] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:42.451982 12858 log.cc:1196] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/1b7c7a7f28df4ca1b618f59cc2a7345f
I20260809 04:50:42.452509 12858 ts_tablet_manager.cc:1939] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 43eea014c2f4402db991371ba0ed7c29: Deleting consensus metadata
I20260809 04:50:42.454365 11980 catalog_manager.cc:4841] TS 43eea014c2f4402db991371ba0ed7c29 (127.31.93.1:41733): tablet 1b7c7a7f28df4ca1b618f59cc2a7345f (table test-table [id=bd3024be63c74232908684e40c99db6e]) successfully deleted
I20260809 04:50:42.458395 12812 tablet_bootstrap.cc:492] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b: No bootstrap required, opened a new log
I20260809 04:50:42.458564 12812 ts_tablet_manager.cc:1397] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.008s
I20260809 04:50:42.459502 12812 raft_consensus.cc:348] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.459771 12812 raft_consensus.cc:374] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:42.459875 12812 raft_consensus.cc:729] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6879888e3a4686870acea2e6d60f2b, State: Initialized, Role: FOLLOWER
I20260809 04:50:42.460189 12812 consensus_queue.cc:260] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.460736 12811 tablet_bootstrap.cc:492] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e: No bootstrap required, opened a new log
I20260809 04:50:42.460875 12811 ts_tablet_manager.cc:1397] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20260809 04:50:42.461615 12811 raft_consensus.cc:348] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.461849 12811 raft_consensus.cc:374] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:42.461951 12811 raft_consensus.cc:729] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d65eb4f9024c81bc2b53669a9e576e, State: Initialized, Role: FOLLOWER
I20260809 04:50:42.462357 12811 consensus_queue.cc:260] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.463524 12812 ts_tablet_manager.cc:1428] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260809 04:50:42.463685 12816 tablet_bootstrap.cc:492] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29: No bootstrap required, opened a new log
I20260809 04:50:42.463877 12816 ts_tablet_manager.cc:1397] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.019s
I20260809 04:50:42.464738 12816 raft_consensus.cc:348] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.465044 12816 raft_consensus.cc:374] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:42.465217 12811 ts_tablet_manager.cc:1428] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260809 04:50:42.465215 12816 raft_consensus.cc:729] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43eea014c2f4402db991371ba0ed7c29, State: Initialized, Role: FOLLOWER
I20260809 04:50:42.465658 12856 ts_tablet_manager.cc:1918] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:42.465821 12856 log.cc:1196] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/1b7c7a7f28df4ca1b618f59cc2a7345f
I20260809 04:50:42.465639 12816 consensus_queue.cc:260] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.466377 12856 ts_tablet_manager.cc:1939] T 1b7c7a7f28df4ca1b618f59cc2a7345f P 2f6879888e3a4686870acea2e6d60f2b: Deleting consensus metadata
I20260809 04:50:42.468161 11984 catalog_manager.cc:4841] TS 2f6879888e3a4686870acea2e6d60f2b (127.31.93.3:39031): tablet 1b7c7a7f28df4ca1b618f59cc2a7345f (table test-table [id=bd3024be63c74232908684e40c99db6e]) successfully deleted
I20260809 04:50:42.468421 12816 ts_tablet_manager.cc:1428] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260809 04:50:42.605747 12852 raft_consensus.cc:482] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:50:42.605970 12852 raft_consensus.cc:504] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.606921 12852 leader_election.cc:288] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43eea014c2f4402db991371ba0ed7c29 (127.31.93.1:41733), 2f6879888e3a4686870acea2e6d60f2b (127.31.93.3:39031)
I20260809 04:50:42.607412 12246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8f2944c3ae4127a8b4fb76c0686573" candidate_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eea014c2f4402db991371ba0ed7c29" is_pre_election: true
I20260809 04:50:42.607432 12688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8f2944c3ae4127a8b4fb76c0686573" candidate_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6879888e3a4686870acea2e6d60f2b" is_pre_election: true
I20260809 04:50:42.607654 12246 raft_consensus.cc:2454] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d65eb4f9024c81bc2b53669a9e576e in term 0.
I20260809 04:50:42.607683 12688 raft_consensus.cc:2454] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d65eb4f9024c81bc2b53669a9e576e in term 0.
I20260809 04:50:42.608075 12365 leader_election.cc:302] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43eea014c2f4402db991371ba0ed7c29, c1d65eb4f9024c81bc2b53669a9e576e; no voters:
I20260809 04:50:42.608356 12852 raft_consensus.cc:2781] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:50:42.608474 12852 raft_consensus.cc:482] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:50:42.608565 12852 raft_consensus.cc:3037] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:42.610855 12852 raft_consensus.cc:504] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.611387 12852 leader_election.cc:288] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [CANDIDATE]: Term 1 election: Requested vote from peers 43eea014c2f4402db991371ba0ed7c29 (127.31.93.1:41733), 2f6879888e3a4686870acea2e6d60f2b (127.31.93.3:39031)
I20260809 04:50:42.611811 12246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8f2944c3ae4127a8b4fb76c0686573" candidate_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eea014c2f4402db991371ba0ed7c29"
I20260809 04:50:42.611879 12688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac8f2944c3ae4127a8b4fb76c0686573" candidate_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6879888e3a4686870acea2e6d60f2b"
I20260809 04:50:42.611982 12246 raft_consensus.cc:3037] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:42.612076 12688 raft_consensus.cc:3037] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:42.614073 12246 raft_consensus.cc:2454] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d65eb4f9024c81bc2b53669a9e576e in term 1.
I20260809 04:50:42.614454 12365 leader_election.cc:302] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43eea014c2f4402db991371ba0ed7c29, c1d65eb4f9024c81bc2b53669a9e576e; no voters:
I20260809 04:50:42.614710 12852 raft_consensus.cc:2781] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:42.614861 12852 raft_consensus.cc:686] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [term 1 LEADER]: Becoming Leader. State: Replica: c1d65eb4f9024c81bc2b53669a9e576e, State: Running, Role: LEADER
I20260809 04:50:42.615005 12688 raft_consensus.cc:2454] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d65eb4f9024c81bc2b53669a9e576e in term 1.
I20260809 04:50:42.615139 12852 consensus_queue.cc:237] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.617698 12006 catalog_manager.cc:5495] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e reported cstate change: term changed from 0 to 1, leader changed from <none> to c1d65eb4f9024c81bc2b53669a9e576e (127.31.93.2). New cstate: current_term: 1 leader_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } health_report { overall_health: UNKNOWN } } }
Aug 09 04:50:42 dist-test-slave-4tdj krb5kdc[11939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251040, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.62@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM
W20260809 04:50:42.668202 12773 tablet.cc:2369] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:50:42.671995 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.656514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41783 (local address 127.0.0.1:54400)
0809 04:50:42.657237 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:42.657258 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:42.657416 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:42.657737 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:42.657742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:42.658737 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:42.659095 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.659110 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.660168 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.660174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.661678 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.661695 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.662415 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.662422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.662587 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.663303 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:42.663327 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:42.665954 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:42.669518 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:42.669528 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:42.669533 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:42.669918 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:42.670344 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:42.670347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:42.670350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:42.670505 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:42.670992 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:42.671000 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:42.671159 (+ 159us) client_negotiation.cc:769] Sending connection context
0809 04:50:42.671301 (+ 142us) client_negotiation.cc:240] Negotiation successful
0809 04:50:42.671547 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":399,"threads_started":1}
I20260809 04:50:42.672011 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.656527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.31.93.62:41783)
0809 04:50:42.656777 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:42.656784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:42.657349 (+ 565us) server_negotiation.cc:407] Connection header received
0809 04:50:42.657524 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:42.657528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:42.657589 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:42.657672 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:42.659235 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.660028 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.661811 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.662198 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.662322 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.666137 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:42.666143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:42.666148 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:42.666187 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:42.669390 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:42.670058 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:42.670064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:42.670067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:42.670216 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:42.670615 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:42.670619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:42.670621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:42.670867 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:42.671012 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:42.671409 (+ 397us) server_negotiation.cc:299] Negotiation successful
0809 04:50:42.671691 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":119}
I20260809 04:50:42.677402 12006 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:54400:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 04:50:42.678736 12006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260809 04:50:42.679891 12333 tablet.cc:2369] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:50:42.681128 12545 tablet.cc:2369] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:50:42.686246 12217 tablet_service.cc:1467] Processing CreateTablet for tablet 538581135e234fa89fb2a43eef58c725 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d23be90362e4fb0bd71fd5134f8c270]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:50:42.686427 12432 tablet_service.cc:1467] Processing CreateTablet for tablet 538581135e234fa89fb2a43eef58c725 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d23be90362e4fb0bd71fd5134f8c270]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:50:42.686836 12217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 538581135e234fa89fb2a43eef58c725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:42.686918 12432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 538581135e234fa89fb2a43eef58c725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:42.694060 12811 tablet_bootstrap.cc:492] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e: Bootstrap starting.
I20260809 04:50:42.694590 12653 tablet_service.cc:1467] Processing CreateTablet for tablet 538581135e234fa89fb2a43eef58c725 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d23be90362e4fb0bd71fd5134f8c270]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 04:50:42.695122 12653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 538581135e234fa89fb2a43eef58c725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:42.695784 12811 tablet_bootstrap.cc:654] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:42.700724 12816 tablet_bootstrap.cc:492] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29: Bootstrap starting.
I20260809 04:50:42.702399 12816 tablet_bootstrap.cc:654] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:42.703363 12812 tablet_bootstrap.cc:492] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b: Bootstrap starting.
I20260809 04:50:42.704952 12812 tablet_bootstrap.cc:654] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:42.705404 12816 tablet_bootstrap.cc:492] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29: No bootstrap required, opened a new log
I20260809 04:50:42.705570 12816 ts_tablet_manager.cc:1397] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260809 04:50:42.706369 12816 raft_consensus.cc:348] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.706637 12816 raft_consensus.cc:374] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:42.706717 12816 raft_consensus.cc:729] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43eea014c2f4402db991371ba0ed7c29, State: Initialized, Role: FOLLOWER
I20260809 04:50:42.706823 12811 tablet_bootstrap.cc:492] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e: No bootstrap required, opened a new log
I20260809 04:50:42.707026 12811 ts_tablet_manager.cc:1397] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260809 04:50:42.707823 12811 raft_consensus.cc:348] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.708148 12811 raft_consensus.cc:374] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:42.708016 12816 consensus_queue.cc:260] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.708417 12811 raft_consensus.cc:729] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d65eb4f9024c81bc2b53669a9e576e, State: Initialized, Role: FOLLOWER
I20260809 04:50:42.708504 12831 tablet_replica.cc:439] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } } }
I20260809 04:50:42.708866 12831 tablet_replica.cc:442] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:50:42.708773 12811 consensus_queue.cc:260] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.709334 12816 ts_tablet_manager.cc:1428] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260809 04:50:42.709409 12852 tablet_replica.cc:439] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } } }
I20260809 04:50:42.709682 12852 tablet_replica.cc:442] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:50:42.710899 12811 ts_tablet_manager.cc:1428] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260809 04:50:42.718973 12812 tablet_bootstrap.cc:492] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b: No bootstrap required, opened a new log
I20260809 04:50:42.719121 12812 ts_tablet_manager.cc:1397] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260809 04:50:42.719786 12812 raft_consensus.cc:348] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.720005 12812 raft_consensus.cc:374] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:42.720072 12812 raft_consensus.cc:729] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6879888e3a4686870acea2e6d60f2b, State: Initialized, Role: FOLLOWER
I20260809 04:50:42.720309 12812 consensus_queue.cc:260] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.721638 12830 tablet_replica.cc:439] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } } }
I20260809 04:50:42.722018 12830 tablet_replica.cc:442] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:50:42.722024 12812 ts_tablet_manager.cc:1428] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260809 04:50:42.840688 12852 raft_consensus.cc:482] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:50:42.840929 12852 raft_consensus.cc:504] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.841687 12852 leader_election.cc:288] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43eea014c2f4402db991371ba0ed7c29 (127.31.93.1:41733), 2f6879888e3a4686870acea2e6d60f2b (127.31.93.3:39031)
I20260809 04:50:42.842247 12688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538581135e234fa89fb2a43eef58c725" candidate_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6879888e3a4686870acea2e6d60f2b" is_pre_election: true
I20260809 04:50:42.842280 12246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538581135e234fa89fb2a43eef58c725" candidate_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eea014c2f4402db991371ba0ed7c29" is_pre_election: true
I20260809 04:50:42.842607 12246 raft_consensus.cc:2454] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d65eb4f9024c81bc2b53669a9e576e in term 0.
I20260809 04:50:42.842592 12688 raft_consensus.cc:2454] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d65eb4f9024c81bc2b53669a9e576e in term 0.
I20260809 04:50:42.843050 12368 leader_election.cc:302] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6879888e3a4686870acea2e6d60f2b, c1d65eb4f9024c81bc2b53669a9e576e; no voters:
I20260809 04:50:42.843372 12852 raft_consensus.cc:2781] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:50:42.843474 12852 raft_consensus.cc:482] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:50:42.843580 12852 raft_consensus.cc:3037] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:42.845717 12852 raft_consensus.cc:504] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.846237 12852 leader_election.cc:288] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [CANDIDATE]: Term 1 election: Requested vote from peers 43eea014c2f4402db991371ba0ed7c29 (127.31.93.1:41733), 2f6879888e3a4686870acea2e6d60f2b (127.31.93.3:39031)
I20260809 04:50:42.846743 12246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538581135e234fa89fb2a43eef58c725" candidate_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eea014c2f4402db991371ba0ed7c29"
I20260809 04:50:42.846741 12688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538581135e234fa89fb2a43eef58c725" candidate_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6879888e3a4686870acea2e6d60f2b"
I20260809 04:50:42.846964 12688 raft_consensus.cc:3037] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:42.846964 12246 raft_consensus.cc:3037] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:42.849766 12688 raft_consensus.cc:2454] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d65eb4f9024c81bc2b53669a9e576e in term 1.
I20260809 04:50:42.849766 12246 raft_consensus.cc:2454] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d65eb4f9024c81bc2b53669a9e576e in term 1.
I20260809 04:50:42.850155 12368 leader_election.cc:302] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6879888e3a4686870acea2e6d60f2b, c1d65eb4f9024c81bc2b53669a9e576e; no voters:
I20260809 04:50:42.850405 12852 raft_consensus.cc:2781] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:42.850567 12852 raft_consensus.cc:686] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [term 1 LEADER]: Becoming Leader. State: Replica: c1d65eb4f9024c81bc2b53669a9e576e, State: Running, Role: LEADER
I20260809 04:50:42.850808 12852 consensus_queue.cc:237] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:42.852239 12825 tablet_replica.cc:439] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e: TxnStatusTablet state changed. Reason: New leader c1d65eb4f9024c81bc2b53669a9e576e. Latest consensus state: current_term: 1 leader_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } } }
I20260809 04:50:42.852478 12825 tablet_replica.cc:442] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:50:42.853039 12884 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 04:50:42.853281 12884 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 04:50:42.853767 12006 catalog_manager.cc:5495] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e reported cstate change: term changed from 0 to 1, leader changed from <none> to c1d65eb4f9024c81bc2b53669a9e576e (127.31.93.2). New cstate: current_term: 1 leader_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } health_report { overall_health: UNKNOWN } } }
I20260809 04:50:42.870870 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.865288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36372 (local address 127.31.93.2:46389)
0809 04:50:42.865433 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:42.865437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:42.865755 (+ 318us) server_negotiation.cc:407] Connection header received
0809 04:50:42.866040 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:42.866044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:42.866108 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:42.866184 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:50:42.866635 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.867374 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.869218 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.869612 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.869702 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.870190 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:50:42.870348 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:50:42.870415 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:42.870581 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 04:50:42.870641 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 04:50:42.871117 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.865300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:46389 (local address 127.0.0.1:36372)
0809 04:50:42.865668 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:42.865690 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:42.865855 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:42.866241 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:42.866247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:42.866260 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:50:42.866480 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.866492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.867499 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.867506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.869086 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.869103 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.869755 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.869762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.869959 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.870029 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:50:42.870442 (+ 413us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:50:42.870464 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 04:50:42.870569 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 04:50:42.870719 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260809 04:50:42.874312 12688 raft_consensus.cc:1261] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Refusing update from remote peer c1d65eb4f9024c81bc2b53669a9e576e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:42.874394 12246 raft_consensus.cc:1261] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Refusing update from remote peer c1d65eb4f9024c81bc2b53669a9e576e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:42.874919 12825 consensus_queue.cc:1035] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:42.875240 12852 consensus_queue.cc:1035] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:42.879105 12830 tablet_replica.cc:439] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b: TxnStatusTablet state changed. Reason: New leader c1d65eb4f9024c81bc2b53669a9e576e. Latest consensus state: current_term: 1 leader_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } } }
I20260809 04:50:42.879112 12831 tablet_replica.cc:439] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29: TxnStatusTablet state changed. Reason: New leader c1d65eb4f9024c81bc2b53669a9e576e. Latest consensus state: current_term: 1 leader_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } } }
I20260809 04:50:42.879350 12830 tablet_replica.cc:442] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:50:42.879356 12831 tablet_replica.cc:442] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:50:42.881232 12825 tablet_replica.cc:439] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } } }
I20260809 04:50:42.881528 12825 tablet_replica.cc:442] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:50:42.883297 12837 tablet_replica.cc:439] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } } }
I20260809 04:50:42.883602 12837 tablet_replica.cc:442] T 538581135e234fa89fb2a43eef58c725 P c1d65eb4f9024c81bc2b53669a9e576e: This TxnStatusTablet replica's current role is: LEADER
I20260809 04:50:42.885560 12830 tablet_replica.cc:439] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } } }
I20260809 04:50:42.885787 12830 tablet_replica.cc:442] T 538581135e234fa89fb2a43eef58c725 P 2f6879888e3a4686870acea2e6d60f2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:50:42.888445 12831 tablet_replica.cc:439] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } } }
I20260809 04:50:42.888674 12831 tablet_replica.cc:442] T 538581135e234fa89fb2a43eef58c725 P 43eea014c2f4402db991371ba0ed7c29: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 04:50:42.903539 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.897426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:39407 (local address 127.31.93.2:46389)
0809 04:50:42.897632 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:42.897638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:42.898174 (+ 536us) server_negotiation.cc:407] Connection header received
0809 04:50:42.898358 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:42.898363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:42.898441 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:42.898543 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:50:42.898991 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.900096 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.901543 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.902240 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.902415 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.902786 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:50:42.902930 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:50:42.903033 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:42.903135 (+ 102us) server_negotiation.cc:299] Negotiation successful
0809 04:50:42.903214 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 04:50:42.903544 12891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:42.897396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:46389 (local address 127.31.93.2:39407)
0809 04:50:42.898078 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:42.898094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:42.898211 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:42.898548 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:42.898551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:42.898560 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 04:50:42.898846 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.898853 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.900209 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.900212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.901380 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:42.901391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:42.902446 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:42.902451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:42.902568 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:42.902616 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 04:50:42.903029 (+ 413us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 04:50:42.903040 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 04:50:42.903098 (+ 58us) client_negotiation.cc:240] Negotiation successful
0809 04:50:42.903194 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":347,"threads_started":1}
I20260809 04:50:42.911401 12688 raft_consensus.cc:1261] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Refusing update from remote peer c1d65eb4f9024c81bc2b53669a9e576e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:42.911700 12246 raft_consensus.cc:1261] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Refusing update from remote peer c1d65eb4f9024c81bc2b53669a9e576e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:42.911917 12852 consensus_queue.cc:1035] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:42.912297 12837 consensus_queue.cc:1035] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:42.918007 12843 mvcc.cc:205] Tried to move back new op lower bound from 7316484271760855040 to 7316484270553923584. Current Snapshot: MvccSnapshot[applied={T|T < 7316484271760855040}]
I20260809 04:50:42.980372 12007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:42.980600 12007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:42.983440 12007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c: Sending DeleteTablet for 3 replicas of tablet ac8f2944c3ae4127a8b4fb76c0686573
I20260809 04:50:42.984232 12432 tablet_service.cc:1514] Processing DeleteTablet for tablet ac8f2944c3ae4127a8b4fb76c0686573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:42 UTC) from {username='oryx', principal='oryx/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:36364
I20260809 04:50:42.984223 12217 tablet_service.cc:1514] Processing DeleteTablet for tablet ac8f2944c3ae4127a8b4fb76c0686573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:42 UTC) from {username='oryx', principal='oryx/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:59258
I20260809 04:50:42.984457 12653 tablet_service.cc:1514] Processing DeleteTablet for tablet ac8f2944c3ae4127a8b4fb76c0686573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:42 UTC) from {username='oryx', principal='oryx/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:60424
I20260809 04:50:42.995175 12900 tablet_replica.cc:330] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b: stopping tablet replica
I20260809 04:50:42.995908 12900 raft_consensus.cc:2229] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:42.996490 12900 raft_consensus.cc:2258] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:42.996522 12899 tablet_replica.cc:330] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e: stopping tablet replica
I20260809 04:50:42.996701 12899 raft_consensus.cc:2229] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:50:42.997126 12899 raft_consensus.cc:2258] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:42.997857 12900 ts_tablet_manager.cc:1905] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:42.999431 12899 ts_tablet_manager.cc:1905] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:43.002871 12898 tablet_replica.cc:330] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29: stopping tablet replica
I20260809 04:50:43.003065 12898 raft_consensus.cc:2229] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:43.003291 12898 raft_consensus.cc:2258] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:43.005286 12900 ts_tablet_manager.cc:1918] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:50:43.005470 12900 log.cc:1196] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/ac8f2944c3ae4127a8b4fb76c0686573
I20260809 04:50:43.006088 12900 ts_tablet_manager.cc:1939] T ac8f2944c3ae4127a8b4fb76c0686573 P 2f6879888e3a4686870acea2e6d60f2b: Deleting consensus metadata
I20260809 04:50:43.006122 12898 ts_tablet_manager.cc:1905] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:43.007257 11984 catalog_manager.cc:4841] TS 2f6879888e3a4686870acea2e6d60f2b (127.31.93.3:39031): tablet ac8f2944c3ae4127a8b4fb76c0686573 (table test-table [id=351a8552c897428d996ddac7f79ec58d]) successfully deleted
I20260809 04:50:43.011267 12898 ts_tablet_manager.cc:1918] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:50:43.011394 12898 log.cc:1196] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/ac8f2944c3ae4127a8b4fb76c0686573
I20260809 04:50:43.011806 12898 ts_tablet_manager.cc:1939] T ac8f2944c3ae4127a8b4fb76c0686573 P 43eea014c2f4402db991371ba0ed7c29: Deleting consensus metadata
I20260809 04:50:43.012843 11980 catalog_manager.cc:4841] TS 43eea014c2f4402db991371ba0ed7c29 (127.31.93.1:41733): tablet ac8f2944c3ae4127a8b4fb76c0686573 (table test-table [id=351a8552c897428d996ddac7f79ec58d]) successfully deleted
I20260809 04:50:43.015188 12899 ts_tablet_manager.cc:1918] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:50:43.015321 12899 log.cc:1196] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/ac8f2944c3ae4127a8b4fb76c0686573
I20260809 04:50:43.015749 12899 ts_tablet_manager.cc:1939] T ac8f2944c3ae4127a8b4fb76c0686573 P c1d65eb4f9024c81bc2b53669a9e576e: Deleting consensus metadata
I20260809 04:50:43.016697 11981 catalog_manager.cc:4841] TS c1d65eb4f9024c81bc2b53669a9e576e (127.31.93.2:46389): tablet ac8f2944c3ae4127a8b4fb76c0686573 (table test-table [id=351a8552c897428d996ddac7f79ec58d]) successfully deleted
Aug 09 04:50:43 dist-test-slave-4tdj krb5kdc[11939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 04:50:43 dist-test-slave-4tdj krb5kdc[11939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:43 dist-test-slave-4tdj krb5kdc[11939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 04:50:43.060124 12150 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:43.048927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.31.93.62:41783)
0809 04:50:43.049129 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:43.049136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:43.049710 (+ 574us) server_negotiation.cc:407] Connection header received
0809 04:50:43.049888 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:43.049892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:43.049958 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:43.050038 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:43.051648 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.052347 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.054173 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.054746 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.054862 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:43.059856 (+ 4994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54406: BlockingRecv error: recv got EOF from 127.0.0.1:54406 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 09 04:50:43 dist-test-slave-4tdj krb5kdc[11939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM
I20260809 04:50:43.076905 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:43.062857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.31.93.62:41783)
0809 04:50:43.063044 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:43.063050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:43.063368 (+ 318us) server_negotiation.cc:407] Connection header received
0809 04:50:43.063498 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:43.063503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:43.063578 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:43.063680 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:43.064715 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.065804 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.067060 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.067678 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.067810 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:43.071105 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:43.071112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:43.071117 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:43.071155 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:43.074368 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:43.074978 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:43.074983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:43.074987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:43.075137 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:43.075439 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:43.075444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:43.075448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:43.075810 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:43.076051 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:43.076292 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 04:50:43.076517 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:50:43.080365 12007 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:50:43.080958 12007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:50:43.087350 12217 tablet_service.cc:1467] Processing CreateTablet for tablet a09a6d7187f646c8a63dd088449776d1 (DEFAULT_TABLE table=test-table [id=2d6c76e799f54a5da5b3a2cdd2265234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:43.087544 12432 tablet_service.cc:1467] Processing CreateTablet for tablet a09a6d7187f646c8a63dd088449776d1 (DEFAULT_TABLE table=test-table [id=2d6c76e799f54a5da5b3a2cdd2265234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:43.087846 12217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a09a6d7187f646c8a63dd088449776d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:43.088030 12432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a09a6d7187f646c8a63dd088449776d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:43.091125 12653 tablet_service.cc:1467] Processing CreateTablet for tablet a09a6d7187f646c8a63dd088449776d1 (DEFAULT_TABLE table=test-table [id=2d6c76e799f54a5da5b3a2cdd2265234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:43.091579 12653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a09a6d7187f646c8a63dd088449776d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:43.094547 12811 tablet_bootstrap.cc:492] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e: Bootstrap starting.
I20260809 04:50:43.095917 12816 tablet_bootstrap.cc:492] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29: Bootstrap starting.
I20260809 04:50:43.096223 12811 tablet_bootstrap.cc:654] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:43.097458 12816 tablet_bootstrap.cc:654] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:43.097785 12812 tablet_bootstrap.cc:492] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b: Bootstrap starting.
I20260809 04:50:43.099293 12812 tablet_bootstrap.cc:654] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:43.108263 12816 tablet_bootstrap.cc:492] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29: No bootstrap required, opened a new log
I20260809 04:50:43.108476 12816 ts_tablet_manager.cc:1397] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20260809 04:50:43.109206 12816 raft_consensus.cc:348] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.109418 12816 raft_consensus.cc:374] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:43.109632 12816 raft_consensus.cc:729] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43eea014c2f4402db991371ba0ed7c29, State: Initialized, Role: FOLLOWER
I20260809 04:50:43.109948 12816 consensus_queue.cc:260] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.110093 12811 tablet_bootstrap.cc:492] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e: No bootstrap required, opened a new log
I20260809 04:50:43.110229 12811 ts_tablet_manager.cc:1397] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.009s
I20260809 04:50:43.111052 12811 raft_consensus.cc:348] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.111407 12811 raft_consensus.cc:374] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:43.111563 12811 raft_consensus.cc:729] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d65eb4f9024c81bc2b53669a9e576e, State: Initialized, Role: FOLLOWER
I20260809 04:50:43.111877 12811 consensus_queue.cc:260] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.112473 12811 ts_tablet_manager.cc:1428] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260809 04:50:43.113499 12816 ts_tablet_manager.cc:1428] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260809 04:50:43.115770 12812 tablet_bootstrap.cc:492] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b: No bootstrap required, opened a new log
I20260809 04:50:43.115928 12812 ts_tablet_manager.cc:1397] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.007s
I20260809 04:50:43.116616 12812 raft_consensus.cc:348] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.116808 12812 raft_consensus.cc:374] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:43.116889 12812 raft_consensus.cc:729] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6879888e3a4686870acea2e6d60f2b, State: Initialized, Role: FOLLOWER
I20260809 04:50:43.117085 12812 consensus_queue.cc:260] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.117583 12812 ts_tablet_manager.cc:1428] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 04:50:43.258925 12831 raft_consensus.cc:482] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:50:43.259120 12831 raft_consensus.cc:504] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.260653 12831 leader_election.cc:288] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1d65eb4f9024c81bc2b53669a9e576e (127.31.93.2:46389), 2f6879888e3a4686870acea2e6d60f2b (127.31.93.3:39031)
I20260809 04:50:43.278579 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:43.261278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:37219 (local address 127.31.93.2:46389)
0809 04:50:43.261446 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:43.261452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:43.271382 (+ 9930us) server_negotiation.cc:407] Connection header received
0809 04:50:43.272262 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:43.272266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:43.272328 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:43.272400 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:43.273164 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.274270 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.276612 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.277535 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.277670 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:43.277778 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:43.278146 (+ 368us) server_negotiation.cc:299] Negotiation successful
0809 04:50:43.278283 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:50:43.278936 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:43.261091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:39031 (local address 127.31.93.1:52201)
0809 04:50:43.271566 (+ 10475us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:43.271584 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:43.271722 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:43.272572 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:43.272577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:43.272621 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:43.273104 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:43.273114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.274167 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.274174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:43.276387 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:43.276408 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.278028 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.278034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:43.278207 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:43.278238 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 04:50:43.278389 (+ 151us) client_negotiation.cc:240] Negotiation successful
0809 04:50:43.278508 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10352,"mutex_wait_us":24,"spinlock_wait_cycles":1920,"thread_start_us":10304,"threads_started":1}
I20260809 04:50:43.279028 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:43.260918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:46389 (local address 127.31.93.1:37219)
0809 04:50:43.271240 (+ 10322us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:43.271266 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:43.272089 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:43.272438 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:43.272442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:43.272467 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:50:43.272954 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:43.272972 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.274421 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.274427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:43.276439 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:43.276449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.277754 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.277761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:43.277925 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:43.277963 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 04:50:43.278100 (+ 137us) client_negotiation.cc:240] Negotiation successful
0809 04:50:43.278247 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10168,"mutex_wait_us":80,"thread_start_us":10019,"threads_started":1}
I20260809 04:50:43.279692 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:43.261121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:52201 (local address 127.31.93.3:39031)
0809 04:50:43.272161 (+ 11040us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:43.272167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:43.272185 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:50:43.272252 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:43.272258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:43.272324 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:43.272409 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:50:43.273295 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.274037 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.276574 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.277800 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.277918 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:43.278046 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:43.279260 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0809 04:50:43.279373 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10888,"thread_start_us":10774,"threads_started":1}
I20260809 04:50:43.280121 12469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a6d7187f646c8a63dd088449776d1" candidate_uuid: "43eea014c2f4402db991371ba0ed7c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" is_pre_election: true
I20260809 04:50:43.280378 12684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a6d7187f646c8a63dd088449776d1" candidate_uuid: "43eea014c2f4402db991371ba0ed7c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6879888e3a4686870acea2e6d60f2b" is_pre_election: true
I20260809 04:50:43.280586 12469 raft_consensus.cc:2454] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43eea014c2f4402db991371ba0ed7c29 in term 0.
I20260809 04:50:43.280658 12684 raft_consensus.cc:2454] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43eea014c2f4402db991371ba0ed7c29 in term 0.
I20260809 04:50:43.281102 12143 leader_election.cc:302] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6879888e3a4686870acea2e6d60f2b, 43eea014c2f4402db991371ba0ed7c29; no voters:
I20260809 04:50:43.281486 12831 raft_consensus.cc:2781] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:50:43.281626 12831 raft_consensus.cc:482] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:50:43.281716 12831 raft_consensus.cc:3037] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:43.284622 12831 raft_consensus.cc:504] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.285315 12831 leader_election.cc:288] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [CANDIDATE]: Term 1 election: Requested vote from peers c1d65eb4f9024c81bc2b53669a9e576e (127.31.93.2:46389), 2f6879888e3a4686870acea2e6d60f2b (127.31.93.3:39031)
I20260809 04:50:43.285703 12684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a6d7187f646c8a63dd088449776d1" candidate_uuid: "43eea014c2f4402db991371ba0ed7c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6879888e3a4686870acea2e6d60f2b"
I20260809 04:50:43.285701 12469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a09a6d7187f646c8a63dd088449776d1" candidate_uuid: "43eea014c2f4402db991371ba0ed7c29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d65eb4f9024c81bc2b53669a9e576e"
I20260809 04:50:43.285887 12469 raft_consensus.cc:3037] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:43.285888 12684 raft_consensus.cc:3037] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:43.287892 12684 raft_consensus.cc:2454] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43eea014c2f4402db991371ba0ed7c29 in term 1.
I20260809 04:50:43.288266 12143 leader_election.cc:302] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6879888e3a4686870acea2e6d60f2b, 43eea014c2f4402db991371ba0ed7c29; no voters:
I20260809 04:50:43.288523 12831 raft_consensus.cc:2781] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:43.289008 12831 raft_consensus.cc:686] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [term 1 LEADER]: Becoming Leader. State: Replica: 43eea014c2f4402db991371ba0ed7c29, State: Running, Role: LEADER
I20260809 04:50:43.289021 12469 raft_consensus.cc:2454] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43eea014c2f4402db991371ba0ed7c29 in term 1.
I20260809 04:50:43.289382 12831 consensus_queue.cc:237] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.292680 12006 catalog_manager.cc:5495] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43eea014c2f4402db991371ba0ed7c29 (127.31.93.1). New cstate: current_term: 1 leader_uuid: "43eea014c2f4402db991371ba0ed7c29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } health_report { overall_health: UNKNOWN } } }
I20260809 04:50:43.339478 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:43.333958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59270 (local address 127.31.93.1:41733)
0809 04:50:43.334575 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:43.334579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:43.334599 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:50:43.334691 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:43.334697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:43.334791 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:43.334958 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:50:43.335272 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.336132 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.337472 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.337888 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.337978 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:43.338391 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:50:43.338946 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:50:43.339058 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:43.339146 (+ 88us) server_negotiation.cc:299] Negotiation successful
0809 04:50:43.339215 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":366,"threads_started":1}
I20260809 04:50:43.343104 12684 raft_consensus.cc:1261] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Refusing update from remote peer 43eea014c2f4402db991371ba0ed7c29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:43.343313 12469 raft_consensus.cc:1261] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e [term 1 FOLLOWER]: Refusing update from remote peer 43eea014c2f4402db991371ba0ed7c29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:43.343832 12929 consensus_queue.cc:1035] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:43.344237 12831 consensus_queue.cc:1035] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:43.358724 12007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:43.358897 12007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:43.361888 12007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c: Sending DeleteTablet for 3 replicas of tablet a09a6d7187f646c8a63dd088449776d1
I20260809 04:50:43.362604 12217 tablet_service.cc:1514] Processing DeleteTablet for tablet a09a6d7187f646c8a63dd088449776d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:43 UTC) from {username='oryx', principal='oryx/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:59258
I20260809 04:50:43.362658 12432 tablet_service.cc:1514] Processing DeleteTablet for tablet a09a6d7187f646c8a63dd088449776d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:43 UTC) from {username='oryx', principal='oryx/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:36364
I20260809 04:50:43.362923 12898 tablet_replica.cc:330] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29: stopping tablet replica
I20260809 04:50:43.362936 12653 tablet_service.cc:1514] Processing DeleteTablet for tablet a09a6d7187f646c8a63dd088449776d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:43 UTC) from {username='oryx', principal='oryx/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:60424
I20260809 04:50:43.363142 12898 raft_consensus.cc:2229] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:50:43.363199 12900 tablet_replica.cc:330] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b: stopping tablet replica
I20260809 04:50:43.363379 12900 raft_consensus.cc:2229] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:43.363591 12900 raft_consensus.cc:2258] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:43.363622 12898 raft_consensus.cc:2258] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:43.363874 12899 tablet_replica.cc:330] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e: stopping tablet replica
I20260809 04:50:43.364065 12899 raft_consensus.cc:2229] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:43.364305 12899 raft_consensus.cc:2258] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:43.364735 12900 ts_tablet_manager.cc:1905] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:43.365900 12007 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 04:50:43.366259 12899 ts_tablet_manager.cc:1905] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 04:50:43.366760 12007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:50:43.369819 12900 ts_tablet_manager.cc:1918] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:43.369922 12900 log.cc:1196] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/a09a6d7187f646c8a63dd088449776d1
I20260809 04:50:43.370177 12900 ts_tablet_manager.cc:1939] T a09a6d7187f646c8a63dd088449776d1 P 2f6879888e3a4686870acea2e6d60f2b: Deleting consensus metadata
I20260809 04:50:43.371116 11984 catalog_manager.cc:4841] TS 2f6879888e3a4686870acea2e6d60f2b (127.31.93.3:39031): tablet a09a6d7187f646c8a63dd088449776d1 (table test-table [id=2d6c76e799f54a5da5b3a2cdd2265234]) successfully deleted
I20260809 04:50:43.371512 12898 ts_tablet_manager.cc:1905] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:43.378075 12898 ts_tablet_manager.cc:1918] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:43.378225 12898 log.cc:1196] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/a09a6d7187f646c8a63dd088449776d1
I20260809 04:50:43.378674 12898 ts_tablet_manager.cc:1939] T a09a6d7187f646c8a63dd088449776d1 P 43eea014c2f4402db991371ba0ed7c29: Deleting consensus metadata
I20260809 04:50:43.380028 11980 catalog_manager.cc:4841] TS 43eea014c2f4402db991371ba0ed7c29 (127.31.93.1:41733): tablet a09a6d7187f646c8a63dd088449776d1 (table test-table [id=2d6c76e799f54a5da5b3a2cdd2265234]) successfully deleted
I20260809 04:50:43.380021 12653 tablet_service.cc:1467] Processing CreateTablet for tablet 9f4f4c508db747d598a06c0d5c577a6e (DEFAULT_TABLE table=test-table [id=753307cc50ff4d84826ede1947e692e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:43.380095 12432 tablet_service.cc:1467] Processing CreateTablet for tablet 9f4f4c508db747d598a06c0d5c577a6e (DEFAULT_TABLE table=test-table [id=753307cc50ff4d84826ede1947e692e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:43.380342 12899 ts_tablet_manager.cc:1918] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:50:43.380406 12653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f4f4c508db747d598a06c0d5c577a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:43.380486 12899 log.cc:1196] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/a09a6d7187f646c8a63dd088449776d1
I20260809 04:50:43.380520 12432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f4f4c508db747d598a06c0d5c577a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:43.380905 12899 ts_tablet_manager.cc:1939] T a09a6d7187f646c8a63dd088449776d1 P c1d65eb4f9024c81bc2b53669a9e576e: Deleting consensus metadata
I20260809 04:50:43.383049 12217 tablet_service.cc:1467] Processing CreateTablet for tablet 9f4f4c508db747d598a06c0d5c577a6e (DEFAULT_TABLE table=test-table [id=753307cc50ff4d84826ede1947e692e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:50:43.383420 12217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f4f4c508db747d598a06c0d5c577a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:50:43.386385 12812 tablet_bootstrap.cc:492] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b: Bootstrap starting.
I20260809 04:50:43.386420 11981 catalog_manager.cc:4841] TS c1d65eb4f9024c81bc2b53669a9e576e (127.31.93.2:46389): tablet a09a6d7187f646c8a63dd088449776d1 (table test-table [id=2d6c76e799f54a5da5b3a2cdd2265234]) successfully deleted
I20260809 04:50:43.386601 12811 tablet_bootstrap.cc:492] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e: Bootstrap starting.
I20260809 04:50:43.387879 12812 tablet_bootstrap.cc:654] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:43.388098 12811 tablet_bootstrap.cc:654] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:43.391780 12816 tablet_bootstrap.cc:492] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29: Bootstrap starting.
I20260809 04:50:43.393477 12816 tablet_bootstrap.cc:654] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:43.398422 12812 tablet_bootstrap.cc:492] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b: No bootstrap required, opened a new log
I20260809 04:50:43.398579 12812 ts_tablet_manager.cc:1397] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260809 04:50:43.399369 12812 raft_consensus.cc:348] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.399638 12812 raft_consensus.cc:374] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:43.399668 12811 tablet_bootstrap.cc:492] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e: No bootstrap required, opened a new log
I20260809 04:50:43.399719 12812 raft_consensus.cc:729] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6879888e3a4686870acea2e6d60f2b, State: Initialized, Role: FOLLOWER
I20260809 04:50:43.399811 12811 ts_tablet_manager.cc:1397] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.012s
I20260809 04:50:43.400027 12812 consensus_queue.cc:260] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.400481 12811 raft_consensus.cc:348] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.400731 12811 raft_consensus.cc:374] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:43.400818 12811 raft_consensus.cc:729] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d65eb4f9024c81bc2b53669a9e576e, State: Initialized, Role: FOLLOWER
I20260809 04:50:43.401060 12811 consensus_queue.cc:260] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.401571 12812 ts_tablet_manager.cc:1428] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 04:50:43.402057 12811 ts_tablet_manager.cc:1428] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260809 04:50:43.406376 12816 tablet_bootstrap.cc:492] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29: No bootstrap required, opened a new log
I20260809 04:50:43.406633 12816 ts_tablet_manager.cc:1397] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.001s
I20260809 04:50:43.407393 12816 raft_consensus.cc:348] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.407667 12816 raft_consensus.cc:374] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:43.407747 12816 raft_consensus.cc:729] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43eea014c2f4402db991371ba0ed7c29, State: Initialized, Role: FOLLOWER
I20260809 04:50:43.407977 12816 consensus_queue.cc:260] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.408483 12816 ts_tablet_manager.cc:1428] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 04:50:43.436833 12837 raft_consensus.cc:482] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:50:43.437028 12837 raft_consensus.cc:504] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.437754 12837 leader_election.cc:288] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43eea014c2f4402db991371ba0ed7c29 (127.31.93.1:41733), 2f6879888e3a4686870acea2e6d60f2b (127.31.93.3:39031)
I20260809 04:50:43.438256 12246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f4f4c508db747d598a06c0d5c577a6e" candidate_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eea014c2f4402db991371ba0ed7c29" is_pre_election: true
I20260809 04:50:43.438283 12684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f4f4c508db747d598a06c0d5c577a6e" candidate_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6879888e3a4686870acea2e6d60f2b" is_pre_election: true
I20260809 04:50:43.438535 12246 raft_consensus.cc:2454] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d65eb4f9024c81bc2b53669a9e576e in term 0.
I20260809 04:50:43.438547 12684 raft_consensus.cc:2454] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d65eb4f9024c81bc2b53669a9e576e in term 0.
I20260809 04:50:43.439059 12368 leader_election.cc:302] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2f6879888e3a4686870acea2e6d60f2b, 43eea014c2f4402db991371ba0ed7c29, c1d65eb4f9024c81bc2b53669a9e576e; no voters:
I20260809 04:50:43.439402 12837 raft_consensus.cc:2781] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:50:43.439508 12837 raft_consensus.cc:482] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:50:43.439597 12837 raft_consensus.cc:3037] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:43.441602 12837 raft_consensus.cc:504] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.442116 12837 leader_election.cc:288] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [CANDIDATE]: Term 1 election: Requested vote from peers 43eea014c2f4402db991371ba0ed7c29 (127.31.93.1:41733), 2f6879888e3a4686870acea2e6d60f2b (127.31.93.3:39031)
I20260809 04:50:43.442574 12246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f4f4c508db747d598a06c0d5c577a6e" candidate_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43eea014c2f4402db991371ba0ed7c29"
I20260809 04:50:43.442687 12684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f4f4c508db747d598a06c0d5c577a6e" candidate_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6879888e3a4686870acea2e6d60f2b"
I20260809 04:50:43.442759 12246 raft_consensus.cc:3037] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:43.442885 12684 raft_consensus.cc:3037] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:43.444871 12246 raft_consensus.cc:2454] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d65eb4f9024c81bc2b53669a9e576e in term 1.
I20260809 04:50:43.445235 12365 leader_election.cc:302] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43eea014c2f4402db991371ba0ed7c29, c1d65eb4f9024c81bc2b53669a9e576e; no voters:
I20260809 04:50:43.445485 12837 raft_consensus.cc:2781] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:43.445622 12837 raft_consensus.cc:686] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [term 1 LEADER]: Becoming Leader. State: Replica: c1d65eb4f9024c81bc2b53669a9e576e, State: Running, Role: LEADER
I20260809 04:50:43.445806 12684 raft_consensus.cc:2454] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d65eb4f9024c81bc2b53669a9e576e in term 1.
I20260809 04:50:43.445885 12837 consensus_queue.cc:237] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } }
I20260809 04:50:43.448406 12007 catalog_manager.cc:5495] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e reported cstate change: term changed from 0 to 1, leader changed from <none> to c1d65eb4f9024c81bc2b53669a9e576e (127.31.93.2). New cstate: current_term: 1 leader_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d65eb4f9024c81bc2b53669a9e576e" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 } health_report { overall_health: UNKNOWN } } }
I20260809 04:50:43.469595 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:43.464197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36382 (local address 127.31.93.2:46389)
0809 04:50:43.464371 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:43.464377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:43.464673 (+ 296us) server_negotiation.cc:407] Connection header received
0809 04:50:43.464844 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:43.464847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:43.464909 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:43.464977 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:50:43.465392 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.466146 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.468058 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.468428 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.468520 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:43.468981 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:50:43.469082 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:50:43.469163 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:43.469283 (+ 120us) server_negotiation.cc:299] Negotiation successful
0809 04:50:43.469343 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 04:50:43.478230 12246 raft_consensus.cc:1261] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Refusing update from remote peer c1d65eb4f9024c81bc2b53669a9e576e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:43.478257 12684 raft_consensus.cc:1261] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Refusing update from remote peer c1d65eb4f9024c81bc2b53669a9e576e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:50:43.479255 12825 consensus_queue.cc:1035] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43eea014c2f4402db991371ba0ed7c29" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:43.479887 12825 consensus_queue.cc:1035] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6879888e3a4686870acea2e6d60f2b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:50:43.536705 12007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:43.536917 12007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:50:43.539830 12007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 095698cca9684c868920b3920294f85c: Sending DeleteTablet for 3 replicas of tablet 9f4f4c508db747d598a06c0d5c577a6e
I20260809 04:50:43.540602 12432 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f4f4c508db747d598a06c0d5c577a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:43 UTC) from {username='oryx', principal='oryx/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:36364
I20260809 04:50:43.540661 12217 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f4f4c508db747d598a06c0d5c577a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:43 UTC) from {username='oryx', principal='oryx/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:59258
I20260809 04:50:43.540817 12653 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f4f4c508db747d598a06c0d5c577a6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:50:43 UTC) from {username='oryx', principal='oryx/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:60424
I20260809 04:50:43.540980 12899 tablet_replica.cc:330] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e: stopping tablet replica
I20260809 04:50:43.541124 12900 tablet_replica.cc:330] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b: stopping tablet replica
I20260809 04:50:43.541160 12899 raft_consensus.cc:2229] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:50:43.541335 12900 raft_consensus.cc:2229] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:43.541508 12899 raft_consensus.cc:2258] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:43.541538 12900 raft_consensus.cc:2258] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:43.541805 12898 tablet_replica.cc:330] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29: stopping tablet replica
I20260809 04:50:43.541987 12898 raft_consensus.cc:2229] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:50:43.542178 12898 raft_consensus.cc:2258] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:50:43.543066 12899 ts_tablet_manager.cc:1905] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:43.543866 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 12116
I20260809 04:50:43.548384 12899 ts_tablet_manager.cc:1918] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:50:43.548513 12899 log.cc:1196] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/9f4f4c508db747d598a06c0d5c577a6e
I20260809 04:50:43.548943 12899 ts_tablet_manager.cc:1939] T 9f4f4c508db747d598a06c0d5c577a6e P c1d65eb4f9024c81bc2b53669a9e576e: Deleting consensus metadata
I20260809 04:50:43.549913 11981 catalog_manager.cc:4841] TS c1d65eb4f9024c81bc2b53669a9e576e (127.31.93.2:46389): tablet 9f4f4c508db747d598a06c0d5c577a6e (table test-table [id=753307cc50ff4d84826ede1947e692e6]) successfully deleted
I20260809 04:50:43.550508 12898 ts_tablet_manager.cc:1905] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:43.555589 12898 ts_tablet_manager.cc:1918] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:50:43.555725 12898 log.cc:1196] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/9f4f4c508db747d598a06c0d5c577a6e
Aug 09 04:50:43 dist-test-slave-4tdj krb5kdc[11939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.93.1@KRBTEST.COM
I20260809 04:50:43.556118 12898 ts_tablet_manager.cc:1939] T 9f4f4c508db747d598a06c0d5c577a6e P 43eea014c2f4402db991371ba0ed7c29: Deleting consensus metadata
I20260809 04:50:43.557013 11980 catalog_manager.cc:4841] TS 43eea014c2f4402db991371ba0ed7c29 (127.31.93.1:41733): tablet 9f4f4c508db747d598a06c0d5c577a6e (table test-table [id=753307cc50ff4d84826ede1947e692e6]) successfully deleted
I20260809 04:50:43.557797 12900 ts_tablet_manager.cc:1905] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:50:43.563925 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:43.546186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.31.93.1:41733)
0809 04:50:43.546393 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:43.546398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:43.546416 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:50:43.546468 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:43.546473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:43.546559 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:43.546660 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:43.548231 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.549338 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.551283 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.551953 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.552088 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:43.558363 (+ 6275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:43.558381 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:43.558398 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:43.558439 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:43.561282 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:43.561899 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:43.561907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:43.561909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:43.562051 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:43.562449 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:43.562454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:43.562456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:43.562807 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:43.563129 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:43.563359 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 04:50:43.563565 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 04:50:43.564380 12900 ts_tablet_manager.cc:1918] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 04:50:43.564509 12900 log.cc:1196] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/9f4f4c508db747d598a06c0d5c577a6e
I20260809 04:50:43.564570 12325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:43.564985 12900 ts_tablet_manager.cc:1939] T 9f4f4c508db747d598a06c0d5c577a6e P 2f6879888e3a4686870acea2e6d60f2b: Deleting consensus metadata
I20260809 04:50:43.566119 11984 catalog_manager.cc:4841] TS 2f6879888e3a4686870acea2e6d60f2b (127.31.93.3:39031): tablet 9f4f4c508db747d598a06c0d5c577a6e (table test-table [id=753307cc50ff4d84826ede1947e692e6]) successfully deleted
I20260809 04:50:43.702566 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 12116
I20260809 04:50:43.727312 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 12340
Aug 09 04:50:43 dist-test-slave-4tdj krb5kdc[11939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.93.2@KRBTEST.COM
I20260809 04:50:43.740767 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:43.727990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.31.93.2:46389)
0809 04:50:43.728210 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:43.728214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:43.728232 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:50:43.728348 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:43.728352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:43.728413 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:43.728499 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:43.729631 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.730351 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.731602 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.732034 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.732128 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:43.736117 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:43.736129 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:43.736143 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:43.736175 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:43.738637 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:43.739182 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:43.739188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:43.739190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:43.739332 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:43.739614 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:43.739617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:43.739619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:43.739913 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:43.740155 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:43.740301 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 04:50:43.740452 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 04:50:43.741390 12537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:43.883512 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 12340
I20260809 04:50:43.911927 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 12548
Aug 09 04:50:43 dist-test-slave-4tdj krb5kdc[11939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.93.3@KRBTEST.COM
I20260809 04:50:43.930732 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:43.912633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.31.93.3:39031)
0809 04:50:43.913110 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:43.913114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:43.913129 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:50:43.913182 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:43.913186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:43.913254 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:43.913345 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:43.914663 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.915383 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.916595 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:43.917008 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:43.917130 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:43.925443 (+ 8313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:43.925458 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:43.925474 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:43.925507 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:43.928096 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:43.928728 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:43.928735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:43.928738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:43.928881 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:43.929227 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:43.929231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:43.929234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:43.929590 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:43.929858 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:43.930174 (+ 316us) server_negotiation.cc:299] Negotiation successful
0809 04:50:43.930415 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":232,"threads_started":1}
I20260809 04:50:43.931432 12763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:44.026376 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 12548
I20260809 04:50:44.049544 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 11961
I20260809 04:50:44.050755 12068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:44.164552 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 11961
2026-08-09T04:50:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5405 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:50:44 dist-test-slave-4tdj krb5kdc[12969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:50:44 dist-test-slave-4tdj krb5kdc[12969](info): set up 2 sockets
Aug 09 04:50:44 dist-test-slave-4tdj krb5kdc[12969](info): commencing operation
krb5kdc: starting...
W20260809 04:50:45.373682 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:50:45 dist-test-slave-4tdj krb5kdc[12969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:50:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:50:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:50:45.585378 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:34179
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:34179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:45.897637 12988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:45.897892 12988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:45.897977 12988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:45.904862 12988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:45.904953 12988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:45.905004 12988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:45.905057 12988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:45.905102 12988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:50:45.914355 12988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:34179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:34179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:45.915424 12988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:45.916577 12988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:45.927417 12996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:45.927506 12999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:45.927464 12997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:45.928468 12988 server_base.cc:1034] running on GCE node
I20260809 04:50:45.929221 12988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:45.930557 12988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:45.931792 12988 hybrid_clock.cc:648] HybridClock initialized: now 1786251045931742 us; error 27 us; skew 500 ppm
I20260809 04:50:45.932124 12988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-09T04:50:46Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1830 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:50:46 dist-test-slave-4tdj krb5kdc[13012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:50:46 dist-test-slave-4tdj krb5kdc[13012](info): set up 2 sockets
Aug 09 04:50:46 dist-test-slave-4tdj krb5kdc[13012](info): commencing operation
krb5kdc: starting...
W20260809 04:50:47.195654 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:50:47 dist-test-slave-4tdj krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:50:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:50:47.405406 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:43475
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:43475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:47.716252 13036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:47.716526 13036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:47.716660 13036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:47.723556 13036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:47.723671 13036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:47.723749 13036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:47.723832 13036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:47.723892 13036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:50:47.733039 13036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:43475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:43475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:47.734192 13036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:47.735437 13036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:47.746145 13046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:47.746244 13042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:47.746212 13048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:47.747213 13036 server_base.cc:1034] running on GCE node
I20260809 04:50:47.747920 13036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:47.749120 13036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:47.750339 13036 hybrid_clock.cc:648] HybridClock initialized: now 1786251047750270 us; error 32 us; skew 500 ppm
I20260809 04:50:47.750654 13036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:47 dist-test-slave-4tdj krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251047, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:47.753437 13036 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:50:47.754829 13036 webserver.cc:460] Webserver started at http://127.31.93.62:45265/ using document root <none> and password file <none>
I20260809 04:50:47.755435 13036 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:47.755542 13036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:47.755836 13036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:47.758556 13036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff"
format_stamp: "Formatted at 2026-08-09 04:50:47 on dist-test-slave-4tdj"
server_key: "9d89402e586e2ac39c257b36a831699b"
server_key_iv: "3879cdf4f6be643c474251d4629cc768"
server_key_version: "encryptionkey@0"
I20260809 04:50:47.759274 13036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff"
format_stamp: "Formatted at 2026-08-09 04:50:47 on dist-test-slave-4tdj"
server_key: "9d89402e586e2ac39c257b36a831699b"
server_key_iv: "3879cdf4f6be643c474251d4629cc768"
server_key_version: "encryptionkey@0"
I20260809 04:50:47.764138 13036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:50:47.767417 13059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:47.768653 13036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260809 04:50:47.768816 13036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff"
format_stamp: "Formatted at 2026-08-09 04:50:47 on dist-test-slave-4tdj"
server_key: "9d89402e586e2ac39c257b36a831699b"
server_key_iv: "3879cdf4f6be643c474251d4629cc768"
server_key_version: "encryptionkey@0"
I20260809 04:50:47.768986 13036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:47.790056 13036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:47.793479 13036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:47.793735 13036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:47.810348 13036 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:43475
I20260809 04:50:47.810424 13130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:43475 every 8 connection(s)
I20260809 04:50:47.811563 13036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:50:47.812181 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 13036
I20260809 04:50:47.812377 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:50:47.813241 32116 external_mini_cluster.cc:1428] Setting key b7a36a04724400e9b60f511c821b43b1
I20260809 04:50:47.816136 13131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 04:50:47 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:47.828047 13131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff: Bootstrap starting.
I20260809 04:50:47.831022 13131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff: Neither blocks nor log segments found. Creating new log.
I20260809 04:50:47.831969 13131 log.cc:824] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:47.834568 13131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff: No bootstrap required, opened a new log
I20260809 04:50:47.834930 13137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:47.815219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47496 (local address 127.31.93.62:43475)
0809 04:50:47.815957 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:47.815972 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:47.816018 (+ 46us) server_negotiation.cc:407] Connection header received
0809 04:50:47.818327 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:47.818354 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:47.818800 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:47.819223 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:47.820492 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:47.822485 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:47.823624 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:47.824575 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:47.824770 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:47.828340 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:47.828349 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:47.828361 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:47.828394 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:47.831180 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:47.831805 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:47.831815 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:47.831821 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:47.831980 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:47.832356 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:47.832360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:47.832362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:47.832888 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:47.833165 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:47.833611 (+ 446us) server_negotiation.cc:299] Negotiation successful
0809 04:50:47.833959 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":315,"threads_started":1}
I20260809 04:50:47.842007 13131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 43475 } }
I20260809 04:50:47.842351 13131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:50:47.842438 13131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4faa2efa0ee04df4b4e5f68b2c5be8ff, State: Initialized, Role: FOLLOWER
I20260809 04:50:47.842922 13131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 43475 } }
I20260809 04:50:47.843125 13131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:50:47.843257 13131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:50:47.843410 13131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:50:47.845400 13131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 43475 } }
I20260809 04:50:47.845841 13131 leader_election.cc:302] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4faa2efa0ee04df4b4e5f68b2c5be8ff; no voters:
I20260809 04:50:47.846496 13131 leader_election.cc:288] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:50:47.846578 13144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:50:47.847342 13144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 1 LEADER]: Becoming Leader. State: Replica: 4faa2efa0ee04df4b4e5f68b2c5be8ff, State: Running, Role: LEADER
I20260809 04:50:47.847818 13144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 43475 } }
I20260809 04:50:47.848471 13131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:47.850260 13145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 43475 } } }
I20260809 04:50:47.850234 13146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4faa2efa0ee04df4b4e5f68b2c5be8ff. Latest consensus state: current_term: 1 leader_uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 43475 } } }
I20260809 04:50:47.850659 13145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:47.850673 13146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:47.852890 13150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:50:47.857013 13150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:50:47.864984 13150 catalog_manager.cc:1347] Generated new cluster ID: 00432dd5960441f1a9f0ae690777317d
I20260809 04:50:47.865098 13150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:50:47.891353 13150 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:50:47.892584 13150 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:50:47.898546 13150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff: Generated new TSK 0
I20260809 04:50:47.899320 13150 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:50:47.951637 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:43475
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:48.264762 13178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:48.265033 13178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:48.265156 13178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:48.272131 13178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:48.272235 13178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:48.272382 13178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:50:48.282055 13178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:43475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:48.283248 13178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:48.284475 13178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:48.296386 13184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:48.296434 13192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:48.296464 13190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:48.297623 13178 server_base.cc:1034] running on GCE node
I20260809 04:50:48.298213 13178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:48.299059 13178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:48.300252 13178 hybrid_clock.cc:648] HybridClock initialized: now 1786251048300205 us; error 39 us; skew 500 ppm
I20260809 04:50:48.300534 13178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:48 dist-test-slave-4tdj krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251048, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:48.303426 13178 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:50:48.304688 13178 webserver.cc:460] Webserver started at http://127.31.93.1:35471/ using document root <none> and password file <none>
I20260809 04:50:48.305325 13178 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:48.305435 13178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:48.305711 13178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:48.308430 13178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "26ecf811af86400fac9eba8d15c1d9c0"
format_stamp: "Formatted at 2026-08-09 04:50:48 on dist-test-slave-4tdj"
server_key: "8fc80403da90a4aa9d011365e5574ee2"
server_key_iv: "8b62bf909904cc180f3cf3f28f1e67a4"
server_key_version: "encryptionkey@0"
I20260809 04:50:48.309149 13178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "26ecf811af86400fac9eba8d15c1d9c0"
format_stamp: "Formatted at 2026-08-09 04:50:48 on dist-test-slave-4tdj"
server_key: "8fc80403da90a4aa9d011365e5574ee2"
server_key_iv: "8b62bf909904cc180f3cf3f28f1e67a4"
server_key_version: "encryptionkey@0"
I20260809 04:50:48.313766 13178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260809 04:50:48.316896 13202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:48.318068 13178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:50:48.318214 13178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "26ecf811af86400fac9eba8d15c1d9c0"
format_stamp: "Formatted at 2026-08-09 04:50:48 on dist-test-slave-4tdj"
server_key: "8fc80403da90a4aa9d011365e5574ee2"
server_key_iv: "8b62bf909904cc180f3cf3f28f1e67a4"
server_key_version: "encryptionkey@0"
I20260809 04:50:48.318413 13178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:48.333338 13178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:48.337155 13178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:48.337404 13178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:48.338263 13178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:48.339602 13178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:48.339681 13178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:48.339809 13178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:48.339862 13178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:48.367378 13178 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:42829
I20260809 04:50:48.367422 13378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:42829 every 8 connection(s)
I20260809 04:50:48.368499 13178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:50:48.372831 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 13178
I20260809 04:50:48.373040 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:50:48.373713 32116 external_mini_cluster.cc:1428] Setting key a5e22e29f0ba8e80b72b394fcf7d64c8
Aug 09 04:50:48 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251048, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:48.388238 13383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:48.370804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:54261 (local address 127.31.93.62:43475)
0809 04:50:48.371253 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:48.371260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:48.372214 (+ 954us) server_negotiation.cc:407] Connection header received
0809 04:50:48.374104 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:48.374111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:48.374193 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:48.374323 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:48.376001 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:48.376697 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:48.378795 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:48.379180 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:48.379270 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:48.382995 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:48.383000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:48.383003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:48.383035 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:48.385422 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:48.386067 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:48.386073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:48.386077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:48.386226 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:48.386637 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:48.386641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:48.386644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:48.386966 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:48.387101 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:48.387697 (+ 596us) server_negotiation.cc:299] Negotiation successful
0809 04:50:48.387906 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":235,"threads_started":1}
I20260809 04:50:48.388880 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:48.371333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:54261)
0809 04:50:48.372055 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:48.372101 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:48.373857 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:48.374591 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:48.374602 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:48.375108 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:48.375811 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:48.375838 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:48.376864 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:48.376869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:48.378671 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:48.378686 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:48.379379 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:48.379385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:48.379634 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:48.380271 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:48.380297 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:48.382782 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:48.385560 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:48.385566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:48.385576 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:48.385949 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:48.386344 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:48.386347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:48.386349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:48.386520 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:48.387071 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:48.387077 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:48.387350 (+ 273us) client_negotiation.cc:769] Sending connection context
0809 04:50:48.387557 (+ 207us) client_negotiation.cc:240] Negotiation successful
0809 04:50:48.387895 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":267,"threads_started":1}
I20260809 04:50:48.390254 13380 heartbeater.cc:347] Connected to a master server at 127.31.93.62:43475
I20260809 04:50:48.390555 13380 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:48.391215 13380 heartbeater.cc:511] Master 127.31.93.62:43475 requested a full tablet report, sending...
I20260809 04:50:48.393170 13082 ts_manager.cc:194] Registered new tserver with Master: 26ecf811af86400fac9eba8d15c1d9c0 (127.31.93.1:42829)
I20260809 04:50:48.394685 13082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:54261
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:50:48.423107 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:43475
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:48.736013 13390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:48.736302 13390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:48.736440 13390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:48.743551 13390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:48.743669 13390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:48.743819 13390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:50:48.753204 13390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:43475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:48.754524 13390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:48.755770 13390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:48.768162 13400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:48.768230 13401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:48.768194 13406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:48.768942 13390 server_base.cc:1034] running on GCE node
I20260809 04:50:48.769481 13390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:48.770385 13390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:48.771582 13390 hybrid_clock.cc:648] HybridClock initialized: now 1786251048771530 us; error 43 us; skew 500 ppm
I20260809 04:50:48.771890 13390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:48 dist-test-slave-4tdj krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251048, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:48.774804 13390 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:50:48.776141 13390 webserver.cc:460] Webserver started at http://127.31.93.2:44017/ using document root <none> and password file <none>
I20260809 04:50:48.776762 13390 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:48.776870 13390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:48.777169 13390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:48.779997 13390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "9a9075434ed74581a5923a8bec41d2d9"
format_stamp: "Formatted at 2026-08-09 04:50:48 on dist-test-slave-4tdj"
server_key: "fc3360c6f6869206312541b11bc9ea68"
server_key_iv: "a470b94bb9ba02f5d1f06fce922275fd"
server_key_version: "encryptionkey@0"
I20260809 04:50:48.780810 13390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "9a9075434ed74581a5923a8bec41d2d9"
format_stamp: "Formatted at 2026-08-09 04:50:48 on dist-test-slave-4tdj"
server_key: "fc3360c6f6869206312541b11bc9ea68"
server_key_iv: "a470b94bb9ba02f5d1f06fce922275fd"
server_key_version: "encryptionkey@0"
I20260809 04:50:48.785889 13390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 04:50:48.789142 13415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:48.790431 13390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:50:48.790593 13390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "9a9075434ed74581a5923a8bec41d2d9"
format_stamp: "Formatted at 2026-08-09 04:50:48 on dist-test-slave-4tdj"
server_key: "fc3360c6f6869206312541b11bc9ea68"
server_key_iv: "a470b94bb9ba02f5d1f06fce922275fd"
server_key_version: "encryptionkey@0"
I20260809 04:50:48.790776 13390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:48.803529 13390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:48.807513 13390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:48.807850 13390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:48.808785 13390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:48.810238 13390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:48.810360 13390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260809 04:50:48.810473 13390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:48.810531 13390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:48.842234 13390 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:45813
I20260809 04:50:48.842280 13589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:45813 every 8 connection(s)
I20260809 04:50:48.843464 13390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:50:48.851583 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 13390
I20260809 04:50:48.851771 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:50:48.852331 32116 external_mini_cluster.cc:1428] Setting key d6194aecdcacb82c1b0f6b9b31e3c042
Aug 09 04:50:48 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251048, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:48.863481 13383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:48.845882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:43317 (local address 127.31.93.62:43475)
0809 04:50:48.846083 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:48.846089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:48.847309 (+ 1220us) server_negotiation.cc:407] Connection header received
0809 04:50:48.849065 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:48.849070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:48.849154 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:48.849237 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:48.850673 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:48.851378 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:48.853453 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:48.854039 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:48.854162 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:48.857986 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:48.857991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:48.857994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:48.858026 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:48.860539 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:48.861359 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:48.861364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:48.861366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:48.861504 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:48.861929 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:48.861932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:48.861934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:48.862171 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:48.862352 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:48.863069 (+ 717us) server_negotiation.cc:299] Negotiation successful
0809 04:50:48.863236 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 04:50:48.864717 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:48.846442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:43317)
0809 04:50:48.847138 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:48.847183 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:48.848820 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:48.849416 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:48.849423 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:48.849933 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:48.850522 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:48.850542 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:48.851527 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:48.851532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:48.853299 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:48.853314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:48.854262 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:48.854271 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:48.854532 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:48.855212 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:48.855241 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:48.857771 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:48.860678 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:48.860686 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:48.860702 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:48.861249 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:48.861605 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:48.861609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:48.861613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:48.861838 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:48.862333 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:48.862340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:48.862717 (+ 377us) client_negotiation.cc:769] Sending connection context
0809 04:50:48.862987 (+ 270us) client_negotiation.cc:240] Negotiation successful
0809 04:50:48.863470 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":265,"threads_started":1}
I20260809 04:50:48.866168 13590 heartbeater.cc:347] Connected to a master server at 127.31.93.62:43475
I20260809 04:50:48.866462 13590 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:48.867128 13590 heartbeater.cc:511] Master 127.31.93.62:43475 requested a full tablet report, sending...
I20260809 04:50:48.868644 13082 ts_manager.cc:194] Registered new tserver with Master: 9a9075434ed74581a5923a8bec41d2d9 (127.31.93.2:45813)
I20260809 04:50:48.869697 13082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:43317
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:50:48.904965 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:43475
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:49.225848 13600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:49.226094 13600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:49.226227 13600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:49.233070 13600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:49.233170 13600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:49.233328 13600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:50:49.242642 13600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:43475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:49.243851 13600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:49.245170 13600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:49.257558 13610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:49.257669 13612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:49.257819 13615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:49.258585 13600 server_base.cc:1034] running on GCE node
I20260809 04:50:49.259133 13600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:49.259984 13600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:49.261181 13600 hybrid_clock.cc:648] HybridClock initialized: now 1786251049261138 us; error 36 us; skew 500 ppm
I20260809 04:50:49.261466 13600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:49 dist-test-slave-4tdj krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251049, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:49.267063 13600 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:50:49.268378 13600 webserver.cc:460] Webserver started at http://127.31.93.3:39699/ using document root <none> and password file <none>
I20260809 04:50:49.269002 13600 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:49.269117 13600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:49.269405 13600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:50:49.272254 13600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "827013bad6154be49fe4abe7222a42a7"
format_stamp: "Formatted at 2026-08-09 04:50:49 on dist-test-slave-4tdj"
server_key: "34f2788c8f2533ceedbad9dd5cc3f4bf"
server_key_iv: "8966cbd2c350e50e413f4c7f9fb40ecd"
server_key_version: "encryptionkey@0"
I20260809 04:50:49.273010 13600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "827013bad6154be49fe4abe7222a42a7"
format_stamp: "Formatted at 2026-08-09 04:50:49 on dist-test-slave-4tdj"
server_key: "34f2788c8f2533ceedbad9dd5cc3f4bf"
server_key_iv: "8966cbd2c350e50e413f4c7f9fb40ecd"
server_key_version: "encryptionkey@0"
I20260809 04:50:49.277822 13600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 04:50:49.281060 13628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:49.282259 13600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:50:49.282454 13600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "827013bad6154be49fe4abe7222a42a7"
format_stamp: "Formatted at 2026-08-09 04:50:49 on dist-test-slave-4tdj"
server_key: "34f2788c8f2533ceedbad9dd5cc3f4bf"
server_key_iv: "8966cbd2c350e50e413f4c7f9fb40ecd"
server_key_version: "encryptionkey@0"
I20260809 04:50:49.282629 13600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:49.300804 13600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:49.304651 13600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:49.304877 13600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:49.305686 13600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:49.306970 13600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:49.307041 13600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:49.307164 13600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:49.307214 13600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:49.335454 13600 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:35241
I20260809 04:50:49.335500 13811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:35241 every 8 connection(s)
I20260809 04:50:49.336658 13600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:50:49.346681 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 13600
I20260809 04:50:49.346835 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:50:49.347249 32116 external_mini_cluster.cc:1428] Setting key 1ed852a6a50f19e4c790f3f776e9de95
Aug 09 04:50:49 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251049, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:50:49.354416 13383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:49.338938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:47899 (local address 127.31.93.62:43475)
0809 04:50:49.339098 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:49.339102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:49.340191 (+ 1089us) server_negotiation.cc:407] Connection header received
0809 04:50:49.341894 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:49.341900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:49.341972 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:49.342049 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:49.343456 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:49.344210 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:49.345732 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:49.346131 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:49.346225 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:49.349550 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:49.349556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:49.349560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:49.349598 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:49.352061 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:49.352663 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:49.352666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:49.352668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:49.352781 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:49.353093 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:49.353096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:49.353098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:49.353343 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:49.353448 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:49.353968 (+ 520us) server_negotiation.cc:299] Negotiation successful
0809 04:50:49.354129 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:50:49.355257 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:49.339336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:47899)
0809 04:50:49.340014 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:49.340050 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:49.341633 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:49.342231 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:49.342238 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:49.342760 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:49.343296 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:49.343313 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:49.344351 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:49.344355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:49.345625 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:49.345638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:49.346332 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:49.346341 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:49.346533 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:49.347142 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:49.347167 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:49.349360 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:50:49.352178 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:49.352184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:49.352194 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:49.352569 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:49.352855 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:50:49.352858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:50:49.352860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:50:49.353018 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:50:49.353431 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:50:49.353436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:50:49.353706 (+ 270us) client_negotiation.cc:769] Sending connection context
0809 04:50:49.353913 (+ 207us) client_negotiation.cc:240] Negotiation successful
0809 04:50:49.354239 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":267,"threads_started":1}
I20260809 04:50:49.356503 13813 heartbeater.cc:347] Connected to a master server at 127.31.93.62:43475
I20260809 04:50:49.356760 13813 heartbeater.cc:464] Registering TS with master...
I20260809 04:50:49.357391 13813 heartbeater.cc:511] Master 127.31.93.62:43475 requested a full tablet report, sending...
I20260809 04:50:49.358695 13082 ts_manager.cc:194] Registered new tserver with Master: 827013bad6154be49fe4abe7222a42a7 (127.31.93.3:35241)
I20260809 04:50:49.359421 13082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:47899
I20260809 04:50:49.362244 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.93.1@KRBTEST.COM" created.
I20260809 04:50:49.397352 13380 heartbeater.cc:503] Master 127.31.93.62:43475 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.1.keytab.
Entry for principal oryx/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.1.keytab.
WARNING: no policy specified for oryx/127.31.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.2.keytab.
Entry for principal oryx/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.2.keytab.
WARNING: no policy specified for oryx/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.3.keytab.
Entry for principal oryx/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.3.keytab.
I20260809 04:50:49.500909 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 13178
Aug 09 04:50:49 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:50:49.515919 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:49.502066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.31.93.1:42829)
0809 04:50:49.502700 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:49.502706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:49.502734 (+ 28us) server_negotiation.cc:407] Connection header received
0809 04:50:49.502798 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:49.502804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:49.503021 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:49.503158 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:49.504423 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:49.505434 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:49.506632 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:49.507177 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:49.507270 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:49.510952 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:49.510963 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:49.510977 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:49.511008 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:49.513644 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:49.514177 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:49.514183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:49.514186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:49.514325 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:49.514625 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:49.514628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:49.514630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:49.514948 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:49.515161 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:49.515378 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 04:50:49.515566 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":324,"threads_started":1}
I20260809 04:50:49.517122 13371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:49.597463 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 13178
I20260809 04:50:49.619479 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 13390
Aug 09 04:50:49 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:50:49.634580 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:49.620347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.31.93.2:45813)
0809 04:50:49.620847 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:49.620853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:49.620878 (+ 25us) server_negotiation.cc:407] Connection header received
0809 04:50:49.620937 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:49.620944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:49.621147 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:49.621299 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:49.622576 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:49.623574 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:49.624826 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:49.625376 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:49.625490 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:49.629475 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:49.629486 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:49.629500 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:49.629533 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:49.632210 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:49.632809 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:49.632816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:49.632818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:49.632940 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:49.633211 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:49.633214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:49.633216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:49.633548 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:49.633790 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:49.633996 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 04:50:49.634188 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":236,"threads_started":1}
I20260809 04:50:49.635754 13585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:49.715656 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 13390
I20260809 04:50:49.733089 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 13600
Aug 09 04:50:49 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:50:49.748831 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:49.734071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57436 (local address 127.31.93.3:35241)
0809 04:50:49.734778 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:49.734786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:49.734815 (+ 29us) server_negotiation.cc:407] Connection header received
0809 04:50:49.734876 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:49.734882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:49.735081 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:49.735205 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:49.736430 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:49.737434 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:49.738792 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:49.739336 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:49.739429 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:49.743541 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:49.743552 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:49.743567 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:49.743604 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:49.746135 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:49.746812 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:49.746819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:49.746822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:49.746956 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:49.747317 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:49.747321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:49.747323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:49.747646 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:49.747960 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:49.748253 (+ 293us) server_negotiation.cc:299] Negotiation successful
0809 04:50:49.748511 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":386,"threads_started":1}
I20260809 04:50:49.750168 13804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:49.830657 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 13600
I20260809 04:50:49.851213 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 13036
I20260809 04:50:49.852484 13124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:50:49.951471 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 13036
I20260809 04:50:49.977355 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:43475
--webserver_interface=127.31.93.62
--webserver_port=45265
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:43475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:50:50.300750 13841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:50.301029 13841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:50.301162 13841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:50.308173 13841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:50:50.308283 13841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:50.308358 13841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:50.308434 13841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:50:50.308493 13841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:50:50.317791 13841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:43475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:43475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=45265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:50.319033 13841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:50.320261 13841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:50.331141 13856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:50.331228 13853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:50.331207 13854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:50.332042 13841 server_base.cc:1034] running on GCE node
I20260809 04:50:50.332782 13841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:50.334035 13841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:50.335237 13841 hybrid_clock.cc:648] HybridClock initialized: now 1786251050335191 us; error 43 us; skew 500 ppm
I20260809 04:50:50.335557 13841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:50 dist-test-slave-4tdj krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251050, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:50.338465 13841 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:50:50.339905 13841 webserver.cc:460] Webserver started at http://127.31.93.62:45265/ using document root <none> and password file <none>
I20260809 04:50:50.340544 13841 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:50.340672 13841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:50.345907 13841 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:50:50.348650 13868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:50.349885 13841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:50:50.350045 13841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff"
format_stamp: "Formatted at 2026-08-09 04:50:47 on dist-test-slave-4tdj"
server_key: "9d89402e586e2ac39c257b36a831699b"
server_key_iv: "3879cdf4f6be643c474251d4629cc768"
server_key_version: "encryptionkey@0"
I20260809 04:50:50.351017 13841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:50.374361 13841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:50.377777 13841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:50.378012 13841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:50.394865 13841 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:43475
I20260809 04:50:50.394901 13954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:43475 every 8 connection(s)
I20260809 04:50:50.396206 13841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:50:50.397589 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 13841
I20260809 04:50:50.398602 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:42829
--local_ip_for_outbound_sockets=127.31.93.1
--tserver_master_addrs=127.31.93.62:43475
--webserver_port=35471
--webserver_interface=127.31.93.1
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.93.1
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:50:50.404819 13955 sys_catalog.cc:263] Verifying existing consensus state
I20260809 04:50:50.408987 13955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff: Bootstrap starting.
I20260809 04:50:50.427155 13955 log.cc:824] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff: Log is configured to *not* fsync() on all Append() calls
I20260809 04:50:50.435423 13955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:50:50.435899 13955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff: Bootstrap complete.
I20260809 04:50:50.444922 13955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 43475 } }
I20260809 04:50:50.445714 13955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4faa2efa0ee04df4b4e5f68b2c5be8ff, State: Initialized, Role: FOLLOWER
I20260809 04:50:50.446380 13955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 43475 } }
I20260809 04:50:50.446619 13955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:50:50.446734 13955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:50:50.446904 13955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 1 FOLLOWER]: Advancing to term 2
I20260809 04:50:50.450492 13955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 43475 } }
I20260809 04:50:50.450949 13955 leader_election.cc:302] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4faa2efa0ee04df4b4e5f68b2c5be8ff; no voters:
I20260809 04:50:50.451506 13955 leader_election.cc:288] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 04:50:50.451646 13959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 2 FOLLOWER]: Leader election won for term 2
I20260809 04:50:50.452790 13959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [term 2 LEADER]: Becoming Leader. State: Replica: 4faa2efa0ee04df4b4e5f68b2c5be8ff, State: Running, Role: LEADER
I20260809 04:50:50.453379 13959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 43475 } }
I20260809 04:50:50.453606 13955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:50:50.455868 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4faa2efa0ee04df4b4e5f68b2c5be8ff. Latest consensus state: current_term: 2 leader_uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 43475 } } }
I20260809 04:50:50.456243 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:50.456744 13960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faa2efa0ee04df4b4e5f68b2c5be8ff" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 43475 } } }
I20260809 04:50:50.457047 13960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff [sys.catalog]: This master's current role is: LEADER
I20260809 04:50:50.463536 13968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:50:50.468581 13968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:50:50.469928 13968 catalog_manager.cc:1259] Loaded cluster ID: 00432dd5960441f1a9f0ae690777317d
I20260809 04:50:50.470072 13968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:50:50.473883 13968 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:50:50.475327 13968 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4faa2efa0ee04df4b4e5f68b2c5be8ff: Loaded TSK: 0
I20260809 04:50:50.476817 13968 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 04:50:50.738114 13957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:50.738453 13957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:50.738631 13957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:50.746107 13957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:50.746214 13957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:50.746405 13957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:50:50.756093 13957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.1.keytab
--principal=oryx/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:42829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=35471
--tserver_master_addrs=127.31.93.62:43475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:50.757351 13957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:50.758719 13957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:50.772744 13995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:50.772802 13997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:50.773423 14000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:50.774179 13957 server_base.cc:1034] running on GCE node
I20260809 04:50:50.774760 13957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:50.775665 13957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:50.776886 13957 hybrid_clock.cc:648] HybridClock initialized: now 1786251050776839 us; error 39 us; skew 500 ppm
I20260809 04:50:50.777173 13957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:50 dist-test-slave-4tdj krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251050, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:50.780138 13957 init.cc:377] Logged in from keytab as oryx/127.31.93.1@KRBTEST.COM (short username oryx)
I20260809 04:50:50.781567 13957 webserver.cc:460] Webserver started at http://127.31.93.1:35471/ using document root <none> and password file <none>
I20260809 04:50:50.782218 13957 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:50.782361 13957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:50.787532 13957 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s
I20260809 04:50:50.790084 14009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:50.791394 13957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:50:50.791532 13957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "26ecf811af86400fac9eba8d15c1d9c0"
format_stamp: "Formatted at 2026-08-09 04:50:48 on dist-test-slave-4tdj"
server_key: "8fc80403da90a4aa9d011365e5574ee2"
server_key_iv: "8b62bf909904cc180f3cf3f28f1e67a4"
server_key_version: "encryptionkey@0"
I20260809 04:50:50.792407 13957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:50.807858 13957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:50.811638 13957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:50.811874 13957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:50.812748 13957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:50.813970 13957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:50.814062 13957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:50.814162 13957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:50.814237 13957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:50.843729 13957 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:42829
I20260809 04:50:50.843788 14204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:42829 every 8 connection(s)
I20260809 04:50:50.845054 13957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:50:50.854228 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 13957
I20260809 04:50:50.854981 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:45813
--local_ip_for_outbound_sockets=127.31.93.2
--tserver_master_addrs=127.31.93.62:43475
--webserver_port=44017
--webserver_interface=127.31.93.2
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.93.2
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:50:50 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:50 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:50.863343 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:50.847830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:53527)
0809 04:50:50.848572 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:50.848610 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:50.850202 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:50.851919 (+ 1717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:50.851930 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:50.852491 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:50.853000 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:50.853016 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:50.855170 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:50.855258 (+ 88us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:50.856489 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:50.856501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:50.857751 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:50.857755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:50.857941 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:50.858607 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:50.858630 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:50.862436 (+ 3806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":302,"threads_started":1}
W20260809 04:50:50.863946 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:50.847854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:53527 (local address 127.31.93.62:43475)
0809 04:50:50.850758 (+ 2904us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:50.850769 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:50.850806 (+ 37us) server_negotiation.cc:407] Connection header received
0809 04:50:50.851081 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:50.851093 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:50.851401 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:50.851722 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:50.853193 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:50.854976 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:50.856640 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:50.857556 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:50.857730 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:50.862836 (+ 5106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:53527: BlockingRecv error: recv got EOF from 127.31.93.1:53527 (error 108)
Metrics: {"server-negotiator.queue_time_us":2701,"thread_start_us":2504,"threads_started":1}
W20260809 04:50:50.864125 14207 heartbeater.cc:644] Failed to heartbeat to 127.31.93.62:43475 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.93.62:43475: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Aug 09 04:50:50 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:50 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:50.872427 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:50.864757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:41023)
0809 04:50:50.864925 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:50.864940 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:50.865015 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:50.865280 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:50.865283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:50.865568 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:50.865705 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:50.865711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:50.866713 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:50.866719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:50.867979 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:50.867995 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:50.868644 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:50.868650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:50.868800 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:50.869323 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:50.869347 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:50.872161 (+ 2814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 04:50:50.872567 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:50.864848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:41023 (local address 127.31.93.62:43475)
0809 04:50:50.864993 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:50.864998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:50.865016 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:50:50.865095 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:50.865100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:50.865166 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:50.865276 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:50.865835 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:50.866576 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:50.868098 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:50.868481 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:50.868565 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:50.872369 (+ 3804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:41023: BlockingRecv error: recv got EOF from 127.31.93.1:41023 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 09 04:50:50 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:50 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:50.880801 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:50.873145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:41109)
0809 04:50:50.873385 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:50.873402 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:50.873496 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:50.873862 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:50.873865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:50.874116 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:50.874260 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:50.874267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:50.875678 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:50.875682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:50.876561 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:50.876571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:50.877433 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:50.877437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:50.877577 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:50.878081 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:50.878098 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:50.880602 (+ 2504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129}
W20260809 04:50:50.881058 14207 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 04:50:50.881376 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:50.873423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:41109 (local address 127.31.93.62:43475)
0809 04:50:50.873607 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:50.873612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:50.873628 (+ 16us) server_negotiation.cc:407] Connection header received
0809 04:50:50.873672 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:50.873677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:50.873753 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:50.873842 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:50.874432 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:50.875435 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:50.876698 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:50.877311 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:50.877442 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:50.880844 (+ 3402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:41109: BlockingRecv error: recv got EOF from 127.31.93.1:41109 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260809 04:50:51.179214 14214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:51.179481 14214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:51.179597 14214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:51.186496 14214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:51.186595 14214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:51.186709 14214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:50:51.195749 14214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.2.keytab
--principal=oryx/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:45813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=44017
--tserver_master_addrs=127.31.93.62:43475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:51.196910 14214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:51.198141 14214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:51.209959 14228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:51.209996 14226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:51.210022 14223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:51.211120 14214 server_base.cc:1034] running on GCE node
I20260809 04:50:51.211632 14214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:51.212433 14214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:51.213624 14214 hybrid_clock.cc:648] HybridClock initialized: now 1786251051213567 us; error 46 us; skew 500 ppm
I20260809 04:50:51.213877 14214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:51 dist-test-slave-4tdj krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251051, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:51.216838 14214 init.cc:377] Logged in from keytab as oryx/127.31.93.2@KRBTEST.COM (short username oryx)
I20260809 04:50:51.218171 14214 webserver.cc:460] Webserver started at http://127.31.93.2:44017/ using document root <none> and password file <none>
I20260809 04:50:51.218806 14214 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:51.218930 14214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:51.223913 14214 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 04:50:51.226418 14238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:51.227609 14214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:50:51.227752 14214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "9a9075434ed74581a5923a8bec41d2d9"
format_stamp: "Formatted at 2026-08-09 04:50:48 on dist-test-slave-4tdj"
server_key: "fc3360c6f6869206312541b11bc9ea68"
server_key_iv: "a470b94bb9ba02f5d1f06fce922275fd"
server_key_version: "encryptionkey@0"
I20260809 04:50:51.228641 14214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:51.250468 14214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:51.254205 14214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:51.254472 14214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:51.255424 14214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:51.256695 14214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:51.256783 14214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:51.256892 14214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:51.256964 14214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:51.286655 14214 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:45813
I20260809 04:50:51.286731 14416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:45813 every 8 connection(s)
I20260809 04:50:51.288057 14214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:50:51.291342 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 14214
I20260809 04:50:51.292093 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:35241
--local_ip_for_outbound_sockets=127.31.93.3
--tserver_master_addrs=127.31.93.62:43475
--webserver_port=39699
--webserver_interface=127.31.93.3
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.93.3
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:50:51 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:51 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 04:50:51.305106 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:51.290535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:53053 (local address 127.31.93.62:43475)
0809 04:50:51.290710 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:51.290714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:51.291964 (+ 1250us) server_negotiation.cc:407] Connection header received
0809 04:50:51.294368 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:51.294373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:51.294443 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:51.294534 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:51.296412 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.297181 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.299197 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.299605 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.299696 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:51.304860 (+ 5164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:53053: BlockingRecv error: recv got EOF from 127.31.93.2:53053 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 04:50:51.305811 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:51.290950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:53053)
0809 04:50:51.291755 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:51.291821 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:51.294055 (+ 2234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:51.294796 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:51.294806 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:51.295475 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:51.296224 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:51.296247 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.297332 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.297338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:51.299059 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:51.299073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.299831 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.299838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:51.300093 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:51.300764 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:51.300796 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:51.304643 (+ 3847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":293,"threads_started":1}
W20260809 04:50:51.306577 14417 heartbeater.cc:644] Failed to heartbeat to 127.31.93.62:43475 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.93.62:43475: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Aug 09 04:50:51 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:51 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:51.314184 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:51.307106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:34513)
0809 04:50:51.307274 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:51.307293 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:51.307384 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:51.307672 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:51.307676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:51.307984 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:51.308106 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:51.308113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.309059 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.309065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:51.309918 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:51.309929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.310634 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.310640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:51.310796 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:51.311288 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:51.311304 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:51.313935 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260809 04:50:51.314414 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:51.307240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:34513 (local address 127.31.93.62:43475)
0809 04:50:51.307416 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:51.307421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:51.307438 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:50:51.307486 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:51.307491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:51.307552 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:51.307627 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:51.308251 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.308926 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.310056 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.310498 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.310607 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:51.314166 (+ 3559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:34513: BlockingRecv error: recv got EOF from 127.31.93.2:34513 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 09 04:50:51 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:51 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:51.321883 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:51.314982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:44351)
0809 04:50:51.315119 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:51.315135 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:51.315260 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:51.315542 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:51.315547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:51.315828 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:51.315961 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:51.315969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.316882 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.316887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:51.317760 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:51.317771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.318426 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.318432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:51.318595 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:51.319082 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:51.319098 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:51.321602 (+ 2504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 04:50:51.321997 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:51.315124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:44351 (local address 127.31.93.62:43475)
0809 04:50:51.315308 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:51.315314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:51.315331 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:50:51.315374 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:51.315378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:51.315426 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:51.315498 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:51.316095 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.316750 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.317896 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.318268 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.318412 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:51.321808 (+ 3396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:44351: BlockingRecv error: recv got EOF from 127.31.93.2:44351 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 04:50:51.322244 14417 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 04:50:51.609452 14422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:50:51.609704 14422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:50:51.609825 14422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:50:51.616539 14422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:50:51.616645 14422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:50:51.616794 14422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:50:51.625692 14422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/krb5kdc/oryx_127.31.93.3.keytab
--principal=oryx/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:35241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=39699
--tserver_master_addrs=127.31.93.62:43475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:50:51.626801 14422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:50:51.627954 14422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:50:51.639926 14431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:51.639916 14432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:50:51.639950 14435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:50:51.640723 14422 server_base.cc:1034] running on GCE node
I20260809 04:50:51.641258 14422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:50:51.642074 14422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:50:51.643251 14422 hybrid_clock.cc:648] HybridClock initialized: now 1786251051643187 us; error 56 us; skew 500 ppm
I20260809 04:50:51.643528 14422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:50:51 dist-test-slave-4tdj krb5kdc[13012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251051, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:50:51.646538 14422 init.cc:377] Logged in from keytab as oryx/127.31.93.3@KRBTEST.COM (short username oryx)
I20260809 04:50:51.647859 14422 webserver.cc:460] Webserver started at http://127.31.93.3:39699/ using document root <none> and password file <none>
I20260809 04:50:51.648478 14422 fs_manager.cc:359] Metadata directory not provided
I20260809 04:50:51.648603 14422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:50:51.653442 14422 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:50:51.656098 14449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:51.657311 14422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 04:50:51.657464 14422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "827013bad6154be49fe4abe7222a42a7"
format_stamp: "Formatted at 2026-08-09 04:50:49 on dist-test-slave-4tdj"
server_key: "34f2788c8f2533ceedbad9dd5cc3f4bf"
server_key_iv: "8966cbd2c350e50e413f4c7f9fb40ecd"
server_key_version: "encryptionkey@0"
I20260809 04:50:51.658382 14422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:50:51.673821 14422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:50:51.677717 14422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:50:51.677955 14422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:50:51.678920 14422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:50:51.680250 14422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:50:51.680352 14422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:51.680459 14422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:50:51.680540 14422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:50:51.708783 14422 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:35241
I20260809 04:50:51.708844 14652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:35241 every 8 connection(s)
I20260809 04:50:51.710103 14422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:50:51.717018 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 14422
Aug 09 04:50:51 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:51 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 04:50:51.725411 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:51.712512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:39515 (local address 127.31.93.62:43475)
0809 04:50:51.712696 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:51.712700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:51.713752 (+ 1052us) server_negotiation.cc:407] Connection header received
0809 04:50:51.715564 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:51.715570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:51.715654 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:51.715743 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:51.717200 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.718217 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.720158 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.720544 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.720668 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:51.725142 (+ 4474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:39515: BlockingRecv error: recv got EOF from 127.31.93.3:39515 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 04:50:51.726094 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:51.712919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:39515)
0809 04:50:51.713588 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:51.713632 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:51.715294 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:51.715987 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:51.715995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:51.716466 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:51.717013 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:51.717031 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.718519 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.718524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:51.720010 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:51.720023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.720768 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.720776 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:51.721000 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:51.721616 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:51.721637 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:51.724929 (+ 3292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":266,"threads_started":1}
W20260809 04:50:51.726836 14653 heartbeater.cc:644] Failed to heartbeat to 127.31.93.62:43475 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.93.62:43475: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
I20260809 04:50:51.733243 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:51.717950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47498 (local address 127.31.93.62:43475)
0809 04:50:51.718695 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:51.718701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:51.718980 (+ 279us) server_negotiation.cc:407] Connection header received
0809 04:50:51.719105 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:51.719111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:51.719183 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:51.719288 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:51.720707 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.721547 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.722839 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.723305 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.723412 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:51.725973 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:50:51.725985 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:50:51.726003 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:50:51.726055 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:50:51.729797 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:51.730469 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:51.730477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:51.730485 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:51.730680 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:50:51.731055 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:50:51.731083 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:50:51.731086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:50:51.731733 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:50:51.731976 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0809 04:50:51.732509 (+ 533us) server_negotiation.cc:299] Negotiation successful
0809 04:50:51.732858 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":631,"thread_start_us":516,"threads_started":1}
Aug 09 04:50:51 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:51 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:51.735354 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:51.727377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:44549)
0809 04:50:51.727536 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:51.727552 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:51.727625 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:51.728054 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:51.728057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:51.728336 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:51.728457 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:51.728464 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.729749 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.729753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:51.730654 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:51.730666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.731420 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.731424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:51.731568 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:51.732072 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:51.732089 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:51.735115 (+ 3026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 04:50:51.735472 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:51.727603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:44549 (local address 127.31.93.62:43475)
0809 04:50:51.727782 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:51.727787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:51.727803 (+ 16us) server_negotiation.cc:407] Connection header received
0809 04:50:51.727845 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:51.727850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:51.727916 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:51.728014 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:51.728569 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.729636 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.730781 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.731295 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.731413 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:51.735265 (+ 3852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:44549: BlockingRecv error: recv got EOF from 127.31.93.3:44549 (error 108)
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":60}
Aug 09 04:50:51 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:51 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:51.743100 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:51.736066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:37085)
0809 04:50:51.736231 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:51.736248 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:51.736338 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:51.736713 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:51.736718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:51.736999 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:51.737134 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:51.737141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.738070 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.738076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:51.738990 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:51.739002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.739641 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.739647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:51.739827 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:51.740244 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:51.740258 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:51.742789 (+ 2531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 04:50:51.743259 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:51.736259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:37085 (local address 127.31.93.62:43475)
0809 04:50:51.736465 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:51.736469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:51.736483 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:50:51.736516 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:51.736519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:51.736580 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:51.736674 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:51.737279 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.737940 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.739126 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.739507 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.739608 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:51.743002 (+ 3394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:37085: BlockingRecv error: recv got EOF from 127.31.93.3:37085 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
W20260809 04:50:51.743373 14653 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 04:50:51 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:51 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:51.889380 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:51.881557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:40343)
0809 04:50:51.882081 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:51.882097 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:51.882207 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:51.882565 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:51.882568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:51.883011 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:51.883201 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:51.883210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.884113 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.884116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:51.885063 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:51.885074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.885807 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.885813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:51.885980 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:51.886605 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:51.886623 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:51.889160 (+ 2537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":281,"threads_started":1}
W20260809 04:50:51.889546 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:51.881644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:40343 (local address 127.31.93.62:43475)
0809 04:50:51.881807 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:51.881811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:51.882167 (+ 356us) server_negotiation.cc:407] Connection header received
0809 04:50:51.882353 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:51.882358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:51.882434 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:51.882537 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:51.883313 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.884007 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.885171 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:51.885645 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:51.885778 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:51.889365 (+ 3587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:40343: BlockingRecv error: recv got EOF from 127.31.93.1:40343 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 09 04:50:52 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:52 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:52.331220 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:52.322808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:47787)
0809 04:50:52.323296 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:52.323317 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:52.323431 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:52.323697 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:52.323700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:52.324131 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:52.324334 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:52.324343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:52.325449 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:52.325455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:52.326618 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:52.326634 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:52.327378 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:52.327385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:52.327550 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:52.328068 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:52.328088 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:52.330918 (+ 2830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":243,"threads_started":1}
W20260809 04:50:52.331388 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:52.322879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:47787 (local address 127.31.93.62:43475)
0809 04:50:52.323087 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:52.323092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:52.323385 (+ 293us) server_negotiation.cc:407] Connection header received
0809 04:50:52.323517 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:52.323522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:52.323588 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:52.323666 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:52.324444 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:52.325294 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:52.326778 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:52.327194 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:52.327303 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:52.331152 (+ 3849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:47787: BlockingRecv error: recv got EOF from 127.31.93.2:47787 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 04:50:52 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:52 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:52.752071 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:52.743946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:45131)
0809 04:50:52.744423 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:52.744438 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:52.744561 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:52.744873 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:52.744878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:52.745359 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:52.745554 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:52.745569 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:52.746706 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:52.746711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:52.747662 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:52.747676 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:52.748371 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:52.748376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:52.748496 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:52.749070 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:52.749088 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:52.751779 (+ 2691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":246,"threads_started":1}
W20260809 04:50:52.752226 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:52.744049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:45131 (local address 127.31.93.62:43475)
0809 04:50:52.744256 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:52.744262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:52.744522 (+ 260us) server_negotiation.cc:407] Connection header received
0809 04:50:52.744678 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:52.744683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:52.744746 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:52.744825 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:52.745715 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:52.746524 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:52.747780 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:52.748217 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:52.748305 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:52.751998 (+ 3693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:45131: BlockingRecv error: recv got EOF from 127.31.93.3:45131 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 04:50:52 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:52 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:52.898029 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:52.890328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:60541)
0809 04:50:52.890821 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:52.890837 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:52.890944 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:52.891293 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:52.891299 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:52.891684 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:52.891831 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:52.891840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:52.892777 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:52.892780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:52.893664 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:52.893674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:52.894276 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:52.894280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:52.894413 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:52.894936 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:52.894953 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:52.897701 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":258,"threads_started":1}
W20260809 04:50:52.898159 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:52.890380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:60541 (local address 127.31.93.62:43475)
0809 04:50:52.890557 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:52.890562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:52.890914 (+ 352us) server_negotiation.cc:407] Connection header received
0809 04:50:52.891058 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:52.891062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:52.891122 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:52.891233 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:52.891947 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:52.892675 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:52.893771 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:52.894163 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:52.894249 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:52.897916 (+ 3667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:60541: BlockingRecv error: recv got EOF from 127.31.93.1:60541 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 09 04:50:53 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:53 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:53.339557 14667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:53.332079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:55619)
0809 04:50:53.332588 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:53.332611 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:53.332737 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:53.333012 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:53.333016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:53.333435 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:53.333605 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:53.333618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:53.334561 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:53.334565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:53.335461 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:53.335472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:53.336070 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:53.336074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:53.336181 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:53.336746 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:53.336764 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:53.339324 (+ 2560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":270,"threads_started":1}
W20260809 04:50:53.339706 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:53.332135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:55619 (local address 127.31.93.62:43475)
0809 04:50:53.332312 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:53.332316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:53.332678 (+ 362us) server_negotiation.cc:407] Connection header received
0809 04:50:53.332825 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:53.332829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:53.332891 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:53.332995 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:53.333741 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:53.334468 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:53.335565 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:53.335958 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:53.336042 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:53.339505 (+ 3463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:55619: BlockingRecv error: recv got EOF from 127.31.93.2:55619 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 09 04:50:53 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:53 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:53.761144 14672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:53.753012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:59939)
0809 04:50:53.753493 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:53.753508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:53.753635 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:53.753979 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:53.753985 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:53.754511 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:53.754663 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:53.754673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:53.755763 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:53.755767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:53.756647 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:53.756661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:53.757374 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:53.757379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:53.757528 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:53.758035 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:53.758053 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:53.760808 (+ 2755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":248,"threads_started":1}
W20260809 04:50:53.761262 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:53.753220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:59939 (local address 127.31.93.62:43475)
0809 04:50:53.753417 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:53.753424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:53.753581 (+ 157us) server_negotiation.cc:407] Connection header received
0809 04:50:53.753747 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:53.753751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:53.753828 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:53.753908 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:53.754826 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:53.755609 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:53.756795 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:53.757219 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:53.757330 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:53.761032 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:59939: BlockingRecv error: recv got EOF from 127.31.93.3:59939 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 09 04:50:53 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:53 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:53.906534 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:53.899049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:44357)
0809 04:50:53.899549 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:53.899566 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:53.899724 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:53.900085 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:53.900088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:53.900421 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:53.900595 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:53.900604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:53.901487 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:53.901490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:53.902409 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:53.902420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:53.903017 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:53.903020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:53.903144 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:53.903656 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:53.903673 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:53.906253 (+ 2580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":269,"threads_started":1}
W20260809 04:50:53.906693 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:53.899098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:44357 (local address 127.31.93.62:43475)
0809 04:50:53.899282 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:53.899289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:53.899665 (+ 376us) server_negotiation.cc:407] Connection header received
0809 04:50:53.899859 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:53.899864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:53.899944 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:53.900045 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:53.900705 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:53.901386 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:53.902526 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:53.902909 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:53.903000 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:53.906463 (+ 3463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:44357: BlockingRecv error: recv got EOF from 127.31.93.1:44357 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 04:50:54 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:54 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:54.351375 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:54.340367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:39643)
0809 04:50:54.340833 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:54.340855 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:54.340982 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:54.341261 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:54.341264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:54.341602 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:54.341777 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:54.341786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:54.342779 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:54.342783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:54.343677 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:54.343689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:54.344301 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:54.344305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:54.344430 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:54.348131 (+ 3701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:54.348158 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:54.351132 (+ 2974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":260,"threads_started":1}
W20260809 04:50:54.351526 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:54.340447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:39643 (local address 127.31.93.62:43475)
0809 04:50:54.340617 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:54.340621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:54.340920 (+ 299us) server_negotiation.cc:407] Connection header received
0809 04:50:54.341090 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:54.341093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:54.341154 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:54.341238 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:54.341887 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:54.342662 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:54.343804 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:54.344195 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:54.344284 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:54.351320 (+ 7036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:39643: BlockingRecv error: recv got EOF from 127.31.93.2:39643 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 09 04:50:54 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:54 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:54.770183 14677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:54.761978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:40535)
0809 04:50:54.762544 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:54.762561 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:54.762696 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:54.763091 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:54.763095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:54.763459 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:54.763644 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:54.763654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:54.764748 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:54.764753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:54.765845 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:54.765859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:54.766540 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:54.766544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:54.766683 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:54.767234 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:54.767251 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:54.769942 (+ 2691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":322,"threads_started":1}
W20260809 04:50:54.770394 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:54.762076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:40535 (local address 127.31.93.62:43475)
0809 04:50:54.762360 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:54.762368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:54.762649 (+ 281us) server_negotiation.cc:407] Connection header received
0809 04:50:54.762812 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:54.762816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:54.762943 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:54.763032 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:54.763793 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:54.764592 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:54.765971 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:54.766428 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:54.766523 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:54.770148 (+ 3625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:40535: BlockingRecv error: recv got EOF from 127.31.93.3:40535 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 09 04:50:54 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:54 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:54.915562 14678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:54.907356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:54591)
0809 04:50:54.907847 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:54.907863 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:54.907993 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:54.908326 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:54.908330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:54.908709 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:54.908857 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:54.908866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:54.909864 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:54.909870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:54.910893 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:54.910907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:54.911637 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:54.911643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:54.911821 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:54.912392 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:54.912410 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:54.915247 (+ 2837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":255,"threads_started":1}
W20260809 04:50:54.915745 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:54.907462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:54591 (local address 127.31.93.62:43475)
0809 04:50:54.907689 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:54.907695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:54.907948 (+ 253us) server_negotiation.cc:407] Connection header received
0809 04:50:54.908117 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:54.908123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:54.908187 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:54.908267 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:54.909006 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:54.909729 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:54.911068 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:54.911491 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:54.911604 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:54.915456 (+ 3852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:54591: BlockingRecv error: recv got EOF from 127.31.93.1:54591 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 09 04:50:55 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:55 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:55.359953 14679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:55.352204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:52751)
0809 04:50:55.352737 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:55.352757 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:55.352876 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:55.353143 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:55.353147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:55.353558 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:55.353731 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:55.353741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:55.354814 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:55.354818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:55.355788 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:55.355801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:55.356415 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:55.356419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:55.356534 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:55.357040 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:55.357056 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:55.359628 (+ 2572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":294,"threads_started":1}
W20260809 04:50:55.360116 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:55.352252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:52751 (local address 127.31.93.62:43475)
0809 04:50:55.352450 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:55.352456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:55.352829 (+ 373us) server_negotiation.cc:407] Connection header received
0809 04:50:55.352967 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:55.352971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:55.353032 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:55.353108 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:55.353856 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:55.354701 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:55.355899 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:55.356311 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:55.356399 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:55.359863 (+ 3464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:52751: BlockingRecv error: recv got EOF from 127.31.93.2:52751 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 09 04:50:55 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:55 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:55.779125 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:55.770993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:35525)
0809 04:50:55.771467 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:55.771483 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:55.771615 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:55.771976 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:55.771980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:55.772351 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:55.772507 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:55.772517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:55.773523 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:55.773527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:55.774558 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:55.774573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:55.775283 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:55.775288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:55.775457 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:55.776039 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:55.776057 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:55.778798 (+ 2741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":256,"threads_started":1}
W20260809 04:50:55.779287 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:55.771064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:35525 (local address 127.31.93.62:43475)
0809 04:50:55.771276 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:55.771282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:55.771567 (+ 285us) server_negotiation.cc:407] Connection header received
0809 04:50:55.771737 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:55.771743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:55.771817 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:55.771901 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:55.772658 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:55.773389 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:55.774718 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:55.775127 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:55.775246 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:55.779006 (+ 3760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:35525: BlockingRecv error: recv got EOF from 127.31.93.3:35525 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 09 04:50:55 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:55 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:55.924134 14681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:55.916417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:42299)
0809 04:50:55.916957 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:55.916974 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:55.917082 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:55.917431 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:55.917436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:55.917953 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:55.918196 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:55.918212 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:55.919199 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:55.919202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:55.920091 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:55.920102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:55.920710 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:55.920714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:55.920832 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:55.921343 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:55.921359 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:55.923889 (+ 2530us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":242,"threads_started":1}
W20260809 04:50:55.924266 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:55.916497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:42299 (local address 127.31.93.62:43475)
0809 04:50:55.916713 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:55.916718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:55.917041 (+ 323us) server_negotiation.cc:407] Connection header received
0809 04:50:55.917190 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:55.917196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:55.917271 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:55.917366 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:55.918381 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:55.919079 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:55.920195 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:55.920605 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:55.920690 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:55.924075 (+ 3385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:42299: BlockingRecv error: recv got EOF from 127.31.93.1:42299 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 09 04:50:56 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:56 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:56.368778 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:56.360884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:44275)
0809 04:50:56.361408 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:56.361429 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:56.361540 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:56.361826 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:56.361830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:56.362278 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:56.362479 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:56.362488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:56.363459 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:56.363462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:56.364350 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:56.364361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:56.364954 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:56.364958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:56.365079 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:56.365609 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:56.365626 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:56.368538 (+ 2912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":274,"threads_started":1}
W20260809 04:50:56.368963 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:56.361051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:44275 (local address 127.31.93.62:43475)
0809 04:50:56.361268 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:56.361274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:56.361501 (+ 227us) server_negotiation.cc:407] Connection header received
0809 04:50:56.361639 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:56.361643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:56.361705 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:56.361781 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:56.362599 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:56.363357 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:56.364456 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:56.364846 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:56.364931 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:56.368729 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:44275: BlockingRecv error: recv got EOF from 127.31.93.2:44275 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 09 04:50:56 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:56 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:56.788069 14684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:56.780197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:41399)
0809 04:50:56.780860 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:56.780879 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:56.781010 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:56.781354 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:56.781358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:56.781917 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:56.782111 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:56.782120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:56.783108 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:56.783112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:56.784001 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:56.784012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:56.784624 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:56.784628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:56.784747 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:56.785279 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:56.785296 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:56.787833 (+ 2537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":392,"threads_started":1}
W20260809 04:50:56.788244 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:56.780262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:41399 (local address 127.31.93.62:43475)
0809 04:50:56.780473 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:56.780480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:56.780970 (+ 490us) server_negotiation.cc:407] Connection header received
0809 04:50:56.781127 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:56.781131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:56.781215 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:56.781312 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:56.782242 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:56.782991 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:56.784107 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:56.784510 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:56.784597 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:56.788007 (+ 3410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:41399: BlockingRecv error: recv got EOF from 127.31.93.3:41399 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 04:50:56 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:56 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:56.932698 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:56.925081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:37945)
0809 04:50:56.925560 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:56.925578 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:56.925705 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:56.925985 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:56.925988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:56.926372 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:56.926525 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:56.926533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:56.927477 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:56.927481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:56.928359 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:56.928369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:56.928992 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:56.928996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:56.929120 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:56.929669 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:56.929687 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:56.932468 (+ 2781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":259,"threads_started":1}
W20260809 04:50:56.932971 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:56.925147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:37945 (local address 127.31.93.62:43475)
0809 04:50:56.925388 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:56.925394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:56.925657 (+ 263us) server_negotiation.cc:407] Connection header received
0809 04:50:56.925801 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:56.925805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:56.925868 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:56.925944 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:56.926642 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:56.927367 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:56.928464 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:56.928864 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:56.928950 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:56.932713 (+ 3763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:37945: BlockingRecv error: recv got EOF from 127.31.93.1:37945 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 09 04:50:57 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:57 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:57.377358 14688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:57.369600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:60537)
0809 04:50:57.370101 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:57.370121 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:57.370234 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:57.370521 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:57.370524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:57.370897 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:57.371043 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:57.371052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:57.372319 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:57.372323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:57.373215 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:57.373227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:57.373840 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:57.373844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:57.373959 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:57.374510 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:57.374527 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:57.377136 (+ 2609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":274,"threads_started":1}
W20260809 04:50:57.377579 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:57.369639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:60537 (local address 127.31.93.62:43475)
0809 04:50:57.369832 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:57.369839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:57.370195 (+ 356us) server_negotiation.cc:407] Connection header received
0809 04:50:57.370347 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:57.370350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:57.370414 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:57.370492 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:57.371203 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:57.372194 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:57.373320 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:57.373728 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:57.373814 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:57.377347 (+ 3533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:60537: BlockingRecv error: recv got EOF from 127.31.93.2:60537 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 09 04:50:57 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:57 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:57.797035 14691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:57.788907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:37863)
0809 04:50:57.789408 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:57.789423 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:57.789550 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:57.789912 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:57.789917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:57.790342 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:57.790505 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:57.790515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:57.791547 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:57.791550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:57.792437 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:57.792449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:57.793329 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:57.793335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:57.793495 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:57.794075 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:57.794109 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:57.796822 (+ 2713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":277,"threads_started":1}
W20260809 04:50:57.797230 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:57.789000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:37863 (local address 127.31.93.62:43475)
0809 04:50:57.789196 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:57.789203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:57.789502 (+ 299us) server_negotiation.cc:407] Connection header received
0809 04:50:57.789672 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:57.789676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:57.789751 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:57.789834 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:57.790648 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:57.791404 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:57.792582 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:57.793169 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:57.793302 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:57.797002 (+ 3700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:37863: BlockingRecv error: recv got EOF from 127.31.93.3:37863 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 09 04:50:57 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:57 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:57.940923 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:57.933496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:38215)
0809 04:50:57.934044 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:57.934061 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:57.934181 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:57.934472 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:57.934476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:57.934872 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:57.935036 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:57.935045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:57.935959 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:57.935962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:57.936868 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:57.936879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:57.937483 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:57.937487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:57.937598 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:57.938109 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:57.938126 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:57.940682 (+ 2556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":261,"threads_started":1}
W20260809 04:50:57.941071 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:57.933572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:38215 (local address 127.31.93.62:43475)
0809 04:50:57.933805 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:57.933812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:57.934155 (+ 343us) server_negotiation.cc:407] Connection header received
0809 04:50:57.934274 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:57.934278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:57.934357 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:57.934433 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:57.935163 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:57.935851 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:57.936972 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:57.937374 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:57.937459 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:57.940875 (+ 3416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:38215: BlockingRecv error: recv got EOF from 127.31.93.1:38215 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 09 04:50:58 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:58 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:58.386567 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:58.378181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:34893)
0809 04:50:58.378746 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:58.378769 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:58.378925 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:58.379322 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:58.379328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:58.379773 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:58.379978 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:58.379992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:58.381244 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:58.381250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:58.382260 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:58.382274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:58.382914 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:58.382919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:58.383030 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:58.383538 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:58.383555 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:58.386187 (+ 2632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":256,"threads_started":1}
W20260809 04:50:58.386646 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:58.378253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:34893 (local address 127.31.93.62:43475)
0809 04:50:58.378493 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:58.378500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:58.378870 (+ 370us) server_negotiation.cc:407] Connection header received
0809 04:50:58.379056 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:58.379062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:58.379155 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:58.379241 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:58.380135 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:58.381104 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:58.382410 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:58.382796 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:58.382885 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:58.386418 (+ 3533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:34893: BlockingRecv error: recv got EOF from 127.31.93.2:34893 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Aug 09 04:50:58 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:58 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:58.806015 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:58.797929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:52893)
0809 04:50:58.798546 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:58.798562 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:58.798684 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:58.798966 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:58.798969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:58.799386 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:58.799558 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:58.799567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:58.800481 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:58.800484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:58.801400 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:58.801411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:58.802039 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:58.802043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:58.802150 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:58.802733 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:58.802750 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:58.805659 (+ 2909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":282,"threads_started":1}
W20260809 04:50:58.806102 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:58.798015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:52893 (local address 127.31.93.62:43475)
0809 04:50:58.798199 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:58.798203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:58.798628 (+ 425us) server_negotiation.cc:407] Connection header received
0809 04:50:58.798773 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:58.798777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:58.798852 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:58.798931 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:58.799679 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:58.800368 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:58.801511 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:58.801921 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:58.802021 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:58.805878 (+ 3857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:52893: BlockingRecv error: recv got EOF from 127.31.93.3:52893 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 09 04:50:58 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:58 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:58.949911 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:58.941852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:53613)
0809 04:50:58.942392 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:58.942408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:58.942528 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:58.942806 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:58.942809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:58.943189 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:58.943340 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:58.943348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:58.944314 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:58.944317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:58.945351 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:58.945366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:58.946153 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:58.946161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:58.946343 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:58.946909 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:58.946926 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:58.949591 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":274,"threads_started":1}
W20260809 04:50:58.950083 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:58.942073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:53613 (local address 127.31.93.62:43475)
0809 04:50:58.942241 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:58.942245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:58.942479 (+ 234us) server_negotiation.cc:407] Connection header received
0809 04:50:58.942622 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:58.942626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:58.942691 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:58.942779 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:58.943475 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:58.944205 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:58.945480 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:58.945961 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:58.946094 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:58.949797 (+ 3703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:53613: BlockingRecv error: recv got EOF from 127.31.93.1:53613 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 04:50:59 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:59 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:59.402483 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:59.387428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:55819)
0809 04:50:59.387968 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:59.387989 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:59.388105 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:59.388380 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:59.388383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:59.388774 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:59.388924 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:59.388932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:59.389983 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:59.389987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:59.390934 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:59.390948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:59.391656 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:59.391661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:59.391792 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:59.399516 (+ 7724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:59.399535 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:59.402218 (+ 2683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":294,"threads_started":1}
W20260809 04:50:59.402642 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:59.387541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:55819 (local address 127.31.93.62:43475)
0809 04:50:59.387712 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:59.387716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:59.388056 (+ 340us) server_negotiation.cc:407] Connection header received
0809 04:50:59.388207 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:59.388211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:59.388272 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:59.388341 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:59.389032 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:59.389868 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:59.391064 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:59.391510 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:59.391610 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:59.402444 (+ 10834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:55819: BlockingRecv error: recv got EOF from 127.31.93.2:55819 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 09 04:50:59 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:59 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:59.815332 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:59.807004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:33105)
0809 04:50:59.807499 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:59.807514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:59.807659 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:59.807980 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:59.807984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:59.808402 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:59.808550 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:59.808560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:59.809575 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:59.809590 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:59.810520 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:59.810533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:59.811302 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:59.811309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:59.811464 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:59.812052 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:59.812070 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:59.815099 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":239,"threads_started":1}
W20260809 04:50:59.815542 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:59.807074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:33105 (local address 127.31.93.62:43475)
0809 04:50:59.807287 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:59.807294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:59.807600 (+ 306us) server_negotiation.cc:407] Connection header received
0809 04:50:59.807776 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:59.807781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:59.807847 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:59.807927 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:59.808709 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:59.809429 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:59.810674 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:59.811118 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:59.811228 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:59.815306 (+ 4078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:33105: BlockingRecv error: recv got EOF from 127.31.93.3:33105 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 04:50:59 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:50:59 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:50:59.958897 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:50:59.950838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:41585)
0809 04:50:59.951443 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0809 04:50:59.951459 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:50:59.951571 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:50:59.951917 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:50:59.951921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:50:59.952313 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:50:59.952474 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:59.952484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:59.953519 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:59.953523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:59.954544 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:50:59.954558 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:50:59.955248 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:50:59.955254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:50:59.955437 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:59.955959 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:50:59.955976 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:50:59.958621 (+ 2645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":364,"threads_started":1}
W20260809 04:50:59.959085 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:50:59.950892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:41585 (local address 127.31.93.62:43475)
0809 04:50:59.951043 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 04:50:59.951048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:50:59.951536 (+ 488us) server_negotiation.cc:407] Connection header received
0809 04:50:59.951705 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:50:59.951709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:50:59.951770 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:50:59.951848 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:50:59.952619 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:59.953385 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:59.954672 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:50:59.955087 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:50:59.955204 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:50:59.958846 (+ 3642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:41585: BlockingRecv error: recv got EOF from 127.31.93.1:41585 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 09 04:51:00 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:00 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:00.411520 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:00.403367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:51623)
0809 04:51:00.403883 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:00.403903 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:00.404050 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:00.404380 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:00.404384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:00.404795 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:00.404959 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:00.404968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:00.406054 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:00.406060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:00.407189 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:00.407202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:00.407902 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:00.407907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:00.408030 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:00.408591 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:00.408610 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:00.411284 (+ 2674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":281,"threads_started":1}
W20260809 04:51:00.411680 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:00.403448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:51623 (local address 127.31.93.62:43475)
0809 04:51:00.403637 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:00.403643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:00.403988 (+ 345us) server_negotiation.cc:407] Connection header received
0809 04:51:00.404180 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:00.404184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:00.404247 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:00.404327 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:00.405107 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:00.405916 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:00.407296 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:00.407765 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:00.407861 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:00.411458 (+ 3597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:51623: BlockingRecv error: recv got EOF from 127.31.93.2:51623 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 09 04:51:00 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:00 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:00.824175 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:00.816173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:52947)
0809 04:51:00.816718 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:00.816735 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:00.816846 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:00.817136 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:00.817139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:00.817532 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:00.817700 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:00.817709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:00.818736 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:00.818740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:00.819657 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:00.819669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:00.820264 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:00.820267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:00.820390 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:00.820929 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:00.820945 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:00.823921 (+ 2976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":285,"threads_started":1}
W20260809 04:51:00.824307 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:00.816275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:52947 (local address 127.31.93.62:43475)
0809 04:51:00.816427 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:00.816431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:00.816799 (+ 368us) server_negotiation.cc:407] Connection header received
0809 04:51:00.816948 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:00.816952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:00.817014 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:00.817092 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:00.817808 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:00.818619 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:00.819762 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:00.820137 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:00.820231 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:00.824104 (+ 3873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:52947: BlockingRecv error: recv got EOF from 127.31.93.3:52947 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 09 04:51:00 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:00 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:00.967664 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:00.959786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:55313)
0809 04:51:00.960294 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:00.960313 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:00.960445 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:00.960814 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:00.960819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:00.961359 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:00.961517 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:00.961527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:00.962600 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:00.962604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:00.963525 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:00.963536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:00.964194 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:00.964198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:00.964307 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:00.964844 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:00.964861 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:00.967440 (+ 2579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":407,"threads_started":1}
W20260809 04:51:00.967850 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:00.959862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:55313 (local address 127.31.93.62:43475)
0809 04:51:00.960172 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:00.960178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:00.960407 (+ 229us) server_negotiation.cc:407] Connection header received
0809 04:51:00.960564 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:00.960568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:00.960648 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:00.960755 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:00.961655 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:00.962487 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:00.963637 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:00.964080 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:00.964175 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:00.967625 (+ 3450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:55313: BlockingRecv error: recv got EOF from 127.31.93.1:55313 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 09 04:51:01 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:01 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:01.420361 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:01.412393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:59941)
0809 04:51:01.412895 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:01.412919 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:01.413037 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:01.413335 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:01.413339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:01.413722 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:01.413892 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:01.413905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:01.415188 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:01.415192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:01.416135 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:01.416148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:01.416748 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:01.416753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:01.416859 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:01.417407 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:01.417426 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:01.420147 (+ 2721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":287,"threads_started":1}
W20260809 04:51:01.420564 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:01.412495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:59941 (local address 127.31.93.62:43475)
0809 04:51:01.412658 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:01.412662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:01.412983 (+ 321us) server_negotiation.cc:407] Connection header received
0809 04:51:01.413131 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:01.413134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:01.413193 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:01.413270 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:01.414027 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:01.415044 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:01.416243 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:01.416636 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:01.416721 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:01.420321 (+ 3600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:59941: BlockingRecv error: recv got EOF from 127.31.93.2:59941 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 09 04:51:01 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:01 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:01.833161 14709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:01.825083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:35125)
0809 04:51:01.825597 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:01.825613 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:01.825740 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:01.826047 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:01.826052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:01.826498 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:01.826667 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:01.826676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:01.827670 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:01.827676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:01.828602 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:01.828614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:01.829299 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:01.829305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:01.829460 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:01.830012 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:01.830029 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:01.832918 (+ 2889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":284,"threads_started":1}
W20260809 04:51:01.833334 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:01.825290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:35125 (local address 127.31.93.62:43475)
0809 04:51:01.825494 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:01.825500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:01.825693 (+ 193us) server_negotiation.cc:407] Connection header received
0809 04:51:01.825847 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:01.825852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:01.825914 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:01.825993 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:01.826816 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:01.827529 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:01.828742 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:01.829150 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:01.829257 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:01.833122 (+ 3865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:35125: BlockingRecv error: recv got EOF from 127.31.93.3:35125 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 09 04:51:01 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:01 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:01.976424 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:01.968489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:40095)
0809 04:51:01.968983 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:01.968999 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:01.969136 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:01.969490 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:01.969495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:01.969916 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:01.970098 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:01.970108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:01.971161 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:01.971165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:01.972028 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:01.972039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:01.972774 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:01.972780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:01.972941 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:01.973459 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:01.973476 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:01.976108 (+ 2632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":265,"threads_started":1}
W20260809 04:51:01.976531 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:01.968569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:40095 (local address 127.31.93.62:43475)
0809 04:51:01.968803 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:01.968809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:01.969081 (+ 272us) server_negotiation.cc:407] Connection header received
0809 04:51:01.969287 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:01.969291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:01.969353 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:01.969434 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:01.970245 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:01.971017 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:01.972182 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:01.972624 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:01.972730 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:01.976309 (+ 3579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:40095: BlockingRecv error: recv got EOF from 127.31.93.1:40095 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 04:51:02 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:02 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:02.428663 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:02.421214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:53211)
0809 04:51:02.421681 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:02.421703 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:02.421814 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:02.422074 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:02.422078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:02.422501 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:02.422695 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:02.422704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:02.423626 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:02.423630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:02.424498 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:02.424509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:02.425129 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:02.425133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:02.425278 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:02.425797 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:02.425814 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:02.428423 (+ 2609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":242,"threads_started":1}
W20260809 04:51:02.428823 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:02.421300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:53211 (local address 127.31.93.62:43475)
0809 04:51:02.421515 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:02.421521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:02.421775 (+ 254us) server_negotiation.cc:407] Connection header received
0809 04:51:02.421902 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:02.421905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:02.421968 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:02.422043 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:02.422799 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:02.423515 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:02.424604 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:02.425015 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:02.425101 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:02.428619 (+ 3518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:53211: BlockingRecv error: recv got EOF from 127.31.93.2:53211 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 09 04:51:02 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:02 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:02.841962 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:02.834087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:47579)
0809 04:51:02.834662 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:02.834679 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:02.834812 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:02.835169 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:02.835174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:02.835673 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:02.835857 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:02.835868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:02.836848 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:02.836851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:02.837793 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:02.837808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:02.838435 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:02.838440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:02.838581 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:02.839130 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:02.839148 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:02.841727 (+ 2579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":333,"threads_started":1}
W20260809 04:51:02.842116 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:02.834164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:47579 (local address 127.31.93.62:43475)
0809 04:51:02.834382 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:02.834388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:02.834763 (+ 375us) server_negotiation.cc:407] Connection header received
0809 04:51:02.834925 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:02.834930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:02.835007 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:02.835107 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:02.835986 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:02.836729 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:02.837901 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:02.838284 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:02.838386 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:02.841903 (+ 3517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:47579: BlockingRecv error: recv got EOF from 127.31.93.3:47579 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 04:51:02 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:02 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:02.987643 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:02.977463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:33017)
0809 04:51:02.978046 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:02.978068 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:02.978236 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:02.978614 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:02.978620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:02.979187 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:02.979417 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:02.979431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:02.980485 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:02.980492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:02.982007 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:02.982026 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:02.982932 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:02.982940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:02.983180 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:02.983831 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:02.983859 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:02.987310 (+ 3451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":287,"threads_started":1}
W20260809 04:51:02.987771 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:02.977529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:33017 (local address 127.31.93.62:43475)
0809 04:51:02.977771 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:02.977778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:02.978146 (+ 368us) server_negotiation.cc:407] Connection header received
0809 04:51:02.978394 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:02.978398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:02.978461 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:02.978536 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:02.979558 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:02.980335 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:02.982142 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:02.982774 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:02.982865 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:02.987522 (+ 4657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:33017: BlockingRecv error: recv got EOF from 127.31.93.1:33017 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 09 04:51:03 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:03 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:03.437516 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:03.429517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:40621)
0809 04:51:03.429998 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:03.430018 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:03.430148 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:03.430537 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:03.430542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:03.430927 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:03.431087 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:03.431096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:03.432166 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:03.432172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:03.433144 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:03.433157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:03.433882 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:03.433888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:03.434055 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:03.434616 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:03.434633 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:03.437243 (+ 2610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":256,"threads_started":1}
W20260809 04:51:03.437678 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:03.429561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:40621 (local address 127.31.93.62:43475)
0809 04:51:03.429751 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:03.429758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:03.430107 (+ 349us) server_negotiation.cc:407] Connection header received
0809 04:51:03.430260 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:03.430265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:03.430385 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:03.430470 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:03.431230 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:03.432024 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:03.433287 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:03.433740 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:03.433847 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:03.437462 (+ 3615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:40621: BlockingRecv error: recv got EOF from 127.31.93.2:40621 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 04:51:03 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:03 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:03.852542 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:03.842812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:55215)
0809 04:51:03.843506 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:03.843527 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:03.843689 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:03.844028 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:03.844033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:03.844615 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:03.844840 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:03.844854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:03.846361 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:03.846368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:03.847736 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:03.847749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:03.848706 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:03.848710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:03.848833 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:03.849438 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:03.849455 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:03.852307 (+ 2852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":407,"threads_started":1}
W20260809 04:51:03.852861 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:03.842891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:55215 (local address 127.31.93.62:43475)
0809 04:51:03.843087 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:03.843093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:03.843622 (+ 529us) server_negotiation.cc:407] Connection header received
0809 04:51:03.843810 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:03.843813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:03.843875 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:03.843944 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:03.845006 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:03.846156 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:03.847888 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:03.848571 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:03.848713 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:03.852587 (+ 3874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:55215: BlockingRecv error: recv got EOF from 127.31.93.3:55215 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 09 04:51:03 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:03 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:03.997671 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:03.988580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:50849)
0809 04:51:03.989101 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:03.989117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:03.989246 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:03.989562 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:03.989567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:03.989967 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:03.990137 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:03.990146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:03.991181 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:03.991188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:03.992105 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:03.992117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:03.992841 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:03.992847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:03.993006 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:03.993671 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:03.993693 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:03.997433 (+ 3740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":254,"threads_started":1}
W20260809 04:51:03.997977 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:03.988645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:50849 (local address 127.31.93.62:43475)
0809 04:51:03.988866 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:03.988873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:03.989200 (+ 327us) server_negotiation.cc:407] Connection header received
0809 04:51:03.989369 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:03.989373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:03.989435 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:03.989511 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:03.990286 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:03.991036 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:03.992247 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:03.992665 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:03.992772 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:03.997700 (+ 4928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:50849: BlockingRecv error: recv got EOF from 127.31.93.1:50849 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 09 04:51:04 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:04 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:04.449980 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:04.438618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:48507)
0809 04:51:04.439147 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:04.439167 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:04.439297 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:04.439606 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:04.439611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:04.440010 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:04.440183 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:04.440193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:04.441212 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:04.441217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:04.442139 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:04.442151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:04.442890 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:04.442896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:04.443042 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:04.447128 (+ 4086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:04.447146 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:04.449648 (+ 2502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":252,"threads_started":1}
W20260809 04:51:04.450119 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:04.438759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:48507 (local address 127.31.93.62:43475)
0809 04:51:04.438954 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:04.438960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:04.439242 (+ 282us) server_negotiation.cc:407] Connection header received
0809 04:51:04.439403 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:04.439409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:04.439486 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:04.439570 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:04.440322 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:04.441074 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:04.442273 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:04.442715 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:04.442823 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:04.449865 (+ 7042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:48507: BlockingRecv error: recv got EOF from 127.31.93.2:48507 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 09 04:51:04 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:04 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:04.861512 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:04.853455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:40791)
0809 04:51:04.854069 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:04.854090 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:04.854232 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:04.854585 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:04.854590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:04.855135 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:04.855303 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:04.855312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:04.856286 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:04.856291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:04.857211 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:04.857224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:04.857946 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:04.857952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:04.858101 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:04.858663 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:04.858682 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:04.861239 (+ 2557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":273,"threads_started":1}
W20260809 04:51:04.861667 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:04.853536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:40791 (local address 127.31.93.62:43475)
0809 04:51:04.853769 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:04.853775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:04.854181 (+ 406us) server_negotiation.cc:407] Connection header received
0809 04:51:04.854363 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:04.854369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:04.854446 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:04.854524 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:04.855450 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:04.856140 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:04.857358 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:04.857773 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:04.857882 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:04.861438 (+ 3556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:40791: BlockingRecv error: recv got EOF from 127.31.93.3:40791 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 09 04:51:04 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:04 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:05.006665 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:04.998635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:50155)
0809 04:51:04.999141 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:04.999161 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:04.999301 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:04.999644 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:04.999657 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:05.000080 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:05.000260 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:05.000270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:05.001310 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:05.001316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:05.002223 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:05.002235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:05.002947 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:05.002954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:05.003122 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:05.003659 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:05.003677 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:05.006287 (+ 2610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":247,"threads_started":1}
W20260809 04:51:05.006776 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:04.998886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:50155 (local address 127.31.93.62:43475)
0809 04:51:04.999162 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:04.999168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:04.999227 (+ 59us) server_negotiation.cc:407] Connection header received
0809 04:51:04.999424 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:04.999429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:04.999505 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:04.999588 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:05.000409 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:05.001170 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:05.002392 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:05.002808 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:05.002918 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:05.006532 (+ 3614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:50155: BlockingRecv error: recv got EOF from 127.31.93.1:50155 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
Aug 09 04:51:05 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:05 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:05.458864 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:05.450930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:58679)
0809 04:51:05.451412 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:05.451432 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:05.451549 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:05.451810 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:05.451812 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:05.452141 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:05.452315 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:05.452324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:05.453249 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:05.453252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:05.454230 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:05.454245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:05.455245 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:05.455251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:05.455431 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:05.455976 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:05.455993 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:05.458622 (+ 2629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":230,"threads_started":1}
W20260809 04:51:05.458981 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:05.451014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:58679 (local address 127.31.93.62:43475)
0809 04:51:05.451181 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:05.451185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:05.451496 (+ 311us) server_negotiation.cc:407] Connection header received
0809 04:51:05.451648 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:05.451652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:05.451712 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:05.451781 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:05.452425 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:05.453141 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:05.454383 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:05.455089 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:05.455215 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:05.458793 (+ 3578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:58679: BlockingRecv error: recv got EOF from 127.31.93.2:58679 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 09 04:51:05 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:05 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.3@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:05.870110 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:05.862323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.3:44659)
0809 04:51:05.862842 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:05.862858 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:05.862968 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:05.863255 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:05.863258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:05.863616 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:05.863819 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:05.863833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:05.864777 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:05.864780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:05.865700 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:05.865712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:05.866402 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:05.866407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:05.866549 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:05.867171 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:05.867196 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:05.869865 (+ 2669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":265,"threads_started":1}
W20260809 04:51:05.870281 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:05.862398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:44659 (local address 127.31.93.62:43475)
0809 04:51:05.862601 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:05.862608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:05.862928 (+ 320us) server_negotiation.cc:407] Connection header received
0809 04:51:05.863073 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:05.863084 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:05.863146 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:05.863222 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:05.863948 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:05.864639 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:05.865802 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:05.866238 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:05.866396 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:05.870075 (+ 3679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.3:44659: BlockingRecv error: recv got EOF from 127.31.93.3:44659 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 04:51:06 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:06 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.1@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:06.015760 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:06.007827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.1:50515)
0809 04:51:06.008312 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:06.008327 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:06.008460 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:06.008784 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:06.008789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:06.009198 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:06.009391 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:06.009401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:06.010473 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:06.010477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:06.011384 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:06.011396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:06.012091 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:06.012097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:06.012263 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:06.012762 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:06.012780 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:06.015488 (+ 2708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":262,"threads_started":1}
W20260809 04:51:06.015897 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:06.007904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:50515 (local address 127.31.93.62:43475)
0809 04:51:06.008097 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:06.008104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:06.008404 (+ 300us) server_negotiation.cc:407] Connection header received
0809 04:51:06.008577 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:06.008582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:06.008652 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:06.008731 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:06.009542 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:06.010335 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:06.011519 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:06.011954 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:06.012063 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:06.015690 (+ 3627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.1:50515: BlockingRecv error: recv got EOF from 127.31.93.1:50515 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 09 04:51:06 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 04:51:06 dist-test-slave-4tdj krb5kdc[13012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.93.2@KRBTEST.COM for oryx/127.31.93.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 04:51:06.468122 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:06.459923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:43475 (local address 127.31.93.2:48605)
0809 04:51:06.460433 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:06.460453 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:06.460577 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:06.460921 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:06.460926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:06.461353 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:06.461545 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:06.461554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:06.462642 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:06.462646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:06.463568 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:06.463580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:06.464283 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:06.464289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:06.464441 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:06.465212 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:06.465242 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:06.467775 (+ 2533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.93.62:43475: Server oryx/127.31.93.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":285,"threads_started":1}
W20260809 04:51:06.468226 14213 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:06.460003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:48605 (local address 127.31.93.62:43475)
0809 04:51:06.460196 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:06.460201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:06.460530 (+ 329us) server_negotiation.cc:407] Connection header received
0809 04:51:06.460711 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:06.460715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:06.460776 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:06.460855 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:06.461693 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:06.462497 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:06.463709 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:06.464146 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:06.464255 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:06.467996 (+ 3741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.93.2:48605: BlockingRecv error: recv got EOF from 127.31.93.2:48605 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 04:51:06.727994 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 13957
I20260809 04:51:06.739243 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:06.728946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35674 (local address 127.31.93.1:42829)
0809 04:51:06.729465 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:06.729472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:06.729498 (+ 26us) server_negotiation.cc:407] Connection header received
0809 04:51:06.729554 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:06.729560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:06.729784 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:06.729930 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:06.731290 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:06.732322 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:06.733498 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:06.734045 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:06.734143 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:06.736376 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:06.736384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:06.736398 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:51:06.736428 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:06.738781 (+ 2353us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.93.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 04:51:06.739017 (+ 236us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35674: Request ticket server kudu/127.31.93.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":236,"threads_started":1}
W20260809 04:51:06.739337 14738 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35674: Request ticket server kudu/127.31.93.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 04:51:06.739343 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 13957: Not authorized: Client connection negotiation failed: client connection to 127.31.93.1:42829: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.93.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 04:51:06.739529 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 13957
I20260809 04:51:06.761715 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 14214
I20260809 04:51:06.773159 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:06.762769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49262 (local address 127.31.93.2:45813)
0809 04:51:06.763369 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:06.763376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:06.763407 (+ 31us) server_negotiation.cc:407] Connection header received
0809 04:51:06.763464 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:06.763470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:06.763683 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:06.763827 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:06.765016 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:06.766037 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:06.767233 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:06.767777 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:06.767869 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:06.770199 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:06.770208 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:06.770222 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:51:06.770262 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:06.772668 (+ 2406us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.93.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 04:51:06.772920 (+ 252us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49262: Request ticket server kudu/127.31.93.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":286,"threads_started":1}
W20260809 04:51:06.773253 14741 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49262: Request ticket server kudu/127.31.93.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 04:51:06.773309 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 14214: Not authorized: Client connection negotiation failed: client connection to 127.31.93.2:45813: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.93.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 04:51:06.773447 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 14214
I20260809 04:51:06.795032 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 14422
I20260809 04:51:06.806452 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:06.795969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.31.93.3:35241)
0809 04:51:06.796514 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:06.796521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:06.796550 (+ 29us) server_negotiation.cc:407] Connection header received
0809 04:51:06.796612 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:06.796619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:06.796853 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:06.797011 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:06.798212 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:06.799234 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:06.800345 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:06.800863 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:06.800961 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:06.803260 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:06.803269 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:06.803282 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:51:06.803317 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:06.805775 (+ 2458us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.93.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 04:51:06.806090 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58720: Request ticket server kudu/127.31.93.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":276,"threads_started":1}
W20260809 04:51:06.806509 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 14422: Not authorized: Client connection negotiation failed: client connection to 127.31.93.3:35241: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.93.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 04:51:06.806555 14742 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58720: Request ticket server kudu/127.31.93.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 04:51:06.806666 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 14422
I20260809 04:51:06.827277 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 13841
I20260809 04:51:06.828383 13947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:51:06.942952 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 13841
2026-08-09T04:51:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20952 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-09T04:51:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:51:07Z Disabled control of system clock
I20260809 04:51:07.027607 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:34361
--webserver_interface=127.31.93.62
--webserver_port=0
--builtin_ntp_servers=127.31.93.20:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:34361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 04:51:07.350759 14757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:51:07.351037 14757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:51:07.351168 14757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:51:07.358100 14757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:51:07.358212 14757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:51:07.358279 14757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:51:07.358374 14757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:51:07.358436 14757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:51:07.367164 14757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:34361
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:34361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:51:07.368391 14757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:51:07.370201 14757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:51:07.381578 14773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:07.381706 14763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:07.381683 14771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:51:07.382256 14757 server_base.cc:1034] running on GCE node
I20260809 04:51:07.383008 14757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:51:07.384282 14757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:51:07.385483 14757 hybrid_clock.cc:648] HybridClock initialized: now 1786251067385437 us; error 35 us; skew 500 ppm
I20260809 04:51:07.385799 14757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:51:07.387887 14757 webserver.cc:460] Webserver started at http://127.31.93.62:38071/ using document root <none> and password file <none>
I20260809 04:51:07.388521 14757 fs_manager.cc:359] Metadata directory not provided
I20260809 04:51:07.388640 14757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:51:07.388948 14757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:51:07.391736 14757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "0e6992a10d1e4f5c8d9636ddab892e3d"
format_stamp: "Formatted at 2026-08-09 04:51:07 on dist-test-slave-4tdj"
server_key: "eb739a2ca5bee34198472a513eaadfdf"
server_key_iv: "9a2175437769b871cef7fd01c6a5eb5d"
server_key_version: "encryptionkey@0"
I20260809 04:51:07.392489 14757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "0e6992a10d1e4f5c8d9636ddab892e3d"
format_stamp: "Formatted at 2026-08-09 04:51:07 on dist-test-slave-4tdj"
server_key: "eb739a2ca5bee34198472a513eaadfdf"
server_key_iv: "9a2175437769b871cef7fd01c6a5eb5d"
server_key_version: "encryptionkey@0"
I20260809 04:51:07.397291 14757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:51:07.400569 14781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:07.401798 14757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 04:51:07.401957 14757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "0e6992a10d1e4f5c8d9636ddab892e3d"
format_stamp: "Formatted at 2026-08-09 04:51:07 on dist-test-slave-4tdj"
server_key: "eb739a2ca5bee34198472a513eaadfdf"
server_key_iv: "9a2175437769b871cef7fd01c6a5eb5d"
server_key_version: "encryptionkey@0"
I20260809 04:51:07.402146 14757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:51:07.423571 14757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:51:07.424291 14757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:51:07.424551 14757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:51:07.441133 14757 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:34361
I20260809 04:51:07.441155 14879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:34361 every 8 connection(s)
I20260809 04:51:07.442386 14757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:51:07.443747 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 14757
I20260809 04:51:07.443940 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:51:07.444756 32116 external_mini_cluster.cc:1428] Setting key c159b0068f94c96bb26d007b1480f5f5
I20260809 04:51:07.446954 14881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:51:07.458127 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:07.446835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60924 (local address 127.31.93.62:34361)
0809 04:51:07.447789 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:07.447799 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:07.447841 (+ 42us) server_negotiation.cc:407] Connection header received
0809 04:51:07.449379 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:07.449401 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:07.449855 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:07.450177 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:07.450838 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:07.452119 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:07.453916 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:07.454596 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:07.454722 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:07.455940 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:07.455950 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:07.455963 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:51:07.456496 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:07.456567 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:51:07.456603 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:07.456703 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:07.456905 (+ 202us) server_negotiation.cc:299] Negotiation successful
0809 04:51:07.457222 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":347,"threads_started":1}
I20260809 04:51:07.459421 14881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d: Bootstrap starting.
I20260809 04:51:07.462512 14881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d: Neither blocks nor log segments found. Creating new log.
I20260809 04:51:07.463478 14881 log.cc:824] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d: Log is configured to *not* fsync() on all Append() calls
I20260809 04:51:07.466090 14881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d: No bootstrap required, opened a new log
I20260809 04:51:07.473202 14881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6992a10d1e4f5c8d9636ddab892e3d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34361 } }
I20260809 04:51:07.473515 14881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:51:07.473589 14881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e6992a10d1e4f5c8d9636ddab892e3d, State: Initialized, Role: FOLLOWER
I20260809 04:51:07.474114 14881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6992a10d1e4f5c8d9636ddab892e3d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34361 } }
I20260809 04:51:07.474349 14881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:51:07.474466 14881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:51:07.474606 14881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:51:07.476590 14881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6992a10d1e4f5c8d9636ddab892e3d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34361 } }
I20260809 04:51:07.477030 14881 leader_election.cc:302] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e6992a10d1e4f5c8d9636ddab892e3d; no voters:
I20260809 04:51:07.477618 14881 leader_election.cc:288] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:51:07.477813 14892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:51:07.478897 14892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [term 1 LEADER]: Becoming Leader. State: Replica: 0e6992a10d1e4f5c8d9636ddab892e3d, State: Running, Role: LEADER
I20260809 04:51:07.479425 14892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6992a10d1e4f5c8d9636ddab892e3d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34361 } }
I20260809 04:51:07.479743 14881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:51:07.482658 14897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e6992a10d1e4f5c8d9636ddab892e3d. Latest consensus state: current_term: 1 leader_uuid: "0e6992a10d1e4f5c8d9636ddab892e3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6992a10d1e4f5c8d9636ddab892e3d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34361 } } }
I20260809 04:51:07.482687 14893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e6992a10d1e4f5c8d9636ddab892e3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6992a10d1e4f5c8d9636ddab892e3d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34361 } } }
I20260809 04:51:07.483053 14897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [sys.catalog]: This master's current role is: LEADER
I20260809 04:51:07.483101 14893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d [sys.catalog]: This master's current role is: LEADER
I20260809 04:51:07.484627 14911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:51:07.490046 14911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:51:07.498814 14911 catalog_manager.cc:1347] Generated new cluster ID: 3f7ee249d7ed4f198e6d2a9bcb053ba7
I20260809 04:51:07.498914 14911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:51:07.520982 14911 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:51:07.522126 14911 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:51:07.529695 14911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e6992a10d1e4f5c8d9636ddab892e3d: Generated new TSK 0
I20260809 04:51:07.530438 14911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:51:07.544061 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34361
--builtin_ntp_servers=127.31.93.20:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260809 04:51:07.894531 14933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:51:07.894786 14933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:51:07.894904 14933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:51:07.901669 14933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:51:07.901753 14933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:51:07.901886 14933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:51:07.910490 14933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:51:07.911664 14933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:51:07.913300 14933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:51:07.925312 14942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:07.925354 14943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:07.925377 14945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:51:07.926378 14933 server_base.cc:1034] running on GCE node
I20260809 04:51:07.926934 14933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:51:07.927790 14933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:51:07.928992 14933 hybrid_clock.cc:648] HybridClock initialized: now 1786251067928940 us; error 42 us; skew 500 ppm
I20260809 04:51:07.929277 14933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:51:07.931321 14933 webserver.cc:460] Webserver started at http://127.31.93.1:41639/ using document root <none> and password file <none>
I20260809 04:51:07.931948 14933 fs_manager.cc:359] Metadata directory not provided
I20260809 04:51:07.932044 14933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:51:07.932343 14933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:51:07.935019 14933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "8f4e891298f94c938b55378304e12d35"
format_stamp: "Formatted at 2026-08-09 04:51:07 on dist-test-slave-4tdj"
server_key: "4180231817f3584436d583693927ec92"
server_key_iv: "529e4014796f3d1e8e823c1e26856e49"
server_key_version: "encryptionkey@0"
I20260809 04:51:07.935868 14933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "8f4e891298f94c938b55378304e12d35"
format_stamp: "Formatted at 2026-08-09 04:51:07 on dist-test-slave-4tdj"
server_key: "4180231817f3584436d583693927ec92"
server_key_iv: "529e4014796f3d1e8e823c1e26856e49"
server_key_version: "encryptionkey@0"
I20260809 04:51:07.940518 14933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 04:51:07.943645 14957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:07.944813 14933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:51:07.944963 14933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "8f4e891298f94c938b55378304e12d35"
format_stamp: "Formatted at 2026-08-09 04:51:07 on dist-test-slave-4tdj"
server_key: "4180231817f3584436d583693927ec92"
server_key_iv: "529e4014796f3d1e8e823c1e26856e49"
server_key_version: "encryptionkey@0"
I20260809 04:51:07.945145 14933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:51:07.959822 14933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:51:07.960506 14933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:51:07.960747 14933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:51:07.961583 14933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:51:07.962875 14933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:51:07.962970 14933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:07.963071 14933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:51:07.963138 14933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:07.992058 14933 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:43385
I20260809 04:51:07.992120 15140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:43385 every 8 connection(s)
I20260809 04:51:07.993208 14933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:51:08.002177 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 14933
I20260809 04:51:08.002357 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:51:08.003196 32116 external_mini_cluster.cc:1428] Setting key 6baa09323dd9726e1cffa943130dc6b8
I20260809 04:51:08.004719 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:07.995478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:47283 (local address 127.31.93.62:34361)
0809 04:51:07.995859 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:07.995863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:07.996996 (+ 1133us) server_negotiation.cc:407] Connection header received
0809 04:51:07.998635 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:07.998640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:07.998704 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:07.998797 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:07.999769 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:08.000475 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:08.001887 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:08.002328 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:08.002437 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:08.003879 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:08.003885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:08.003889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:51:08.003910 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:08.003956 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:51:08.003972 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:08.004052 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:08.004286 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 04:51:08.004372 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":181,"threads_started":1}
I20260809 04:51:08.005689 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:07.996033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:34361 (local address 127.31.93.1:47283)
0809 04:51:07.996853 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:07.996886 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:07.998416 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:07.998975 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:07.998982 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:07.999068 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:07.999590 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:07.999606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:08.000594 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:08.000597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:08.001766 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:08.001781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:08.002565 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:08.002574 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:08.002843 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:08.003544 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:51:08.003548 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:08.003595 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:51:08.003680 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:51:08.004080 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:51:08.004092 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:51:08.004099 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 04:51:08.004267 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 04:51:08.004518 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":257,"threads_started":1}
I20260809 04:51:08.006244 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34361
--builtin_ntp_servers=127.31.93.20:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 04:51:08.007382 15141 heartbeater.cc:347] Connected to a master server at 127.31.93.62:34361
I20260809 04:51:08.007670 15141 heartbeater.cc:464] Registering TS with master...
I20260809 04:51:08.008406 15141 heartbeater.cc:511] Master 127.31.93.62:34361 requested a full tablet report, sending...
I20260809 04:51:08.010326 14816 ts_manager.cc:194] Registered new tserver with Master: 8f4e891298f94c938b55378304e12d35 (127.31.93.1:43385)
I20260809 04:51:08.011776 14816 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.93.1:47283
W20260809 04:51:08.321576 15151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:51:08.321846 15151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:51:08.321966 15151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:51:08.328933 15151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:51:08.329030 15151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:51:08.329169 15151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:51:08.337954 15151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:51:08.339179 15151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:51:08.340885 15151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:51:08.353003 15158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:08.353017 15162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:08.353052 15157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:51:08.354192 15151 server_base.cc:1034] running on GCE node
I20260809 04:51:08.354777 15151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:51:08.355656 15151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:51:08.356843 15151 hybrid_clock.cc:648] HybridClock initialized: now 1786251068356799 us; error 41 us; skew 500 ppm
I20260809 04:51:08.357124 15151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:51:08.359215 15151 webserver.cc:460] Webserver started at http://127.31.93.2:38885/ using document root <none> and password file <none>
I20260809 04:51:08.359834 15151 fs_manager.cc:359] Metadata directory not provided
I20260809 04:51:08.359925 15151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:51:08.360235 15151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:51:08.362870 15151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "92bb3ac16bd147c09ff0a5c7fc832b8c"
format_stamp: "Formatted at 2026-08-09 04:51:08 on dist-test-slave-4tdj"
server_key: "6b3bfb7ea116a44df601671a85758e0a"
server_key_iv: "8c97a20bda918a825017e8c3907c9412"
server_key_version: "encryptionkey@0"
I20260809 04:51:08.363600 15151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "92bb3ac16bd147c09ff0a5c7fc832b8c"
format_stamp: "Formatted at 2026-08-09 04:51:08 on dist-test-slave-4tdj"
server_key: "6b3bfb7ea116a44df601671a85758e0a"
server_key_iv: "8c97a20bda918a825017e8c3907c9412"
server_key_version: "encryptionkey@0"
I20260809 04:51:08.368398 15151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260809 04:51:08.371665 15169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:08.372977 15151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:51:08.373111 15151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "92bb3ac16bd147c09ff0a5c7fc832b8c"
format_stamp: "Formatted at 2026-08-09 04:51:08 on dist-test-slave-4tdj"
server_key: "6b3bfb7ea116a44df601671a85758e0a"
server_key_iv: "8c97a20bda918a825017e8c3907c9412"
server_key_version: "encryptionkey@0"
I20260809 04:51:08.373299 15151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:51:08.398607 15151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:51:08.399322 15151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:51:08.399569 15151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:51:08.400446 15151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:51:08.401757 15151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:51:08.401849 15151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:08.401952 15151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:51:08.402025 15151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:08.431506 15151 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:44835
I20260809 04:51:08.431571 15354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:44835 every 8 connection(s)
I20260809 04:51:08.432721 15151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:51:08.440033 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 15151
I20260809 04:51:08.440172 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:51:08.440675 32116 external_mini_cluster.cc:1428] Setting key 4111d1548b3c8e67dc2b4d30af5fa420
I20260809 04:51:08.443676 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34361
--builtin_ntp_servers=127.31.93.20:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 04:51:08.445259 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:08.435143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:34223 (local address 127.31.93.62:34361)
0809 04:51:08.435320 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:08.435325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:08.436613 (+ 1288us) server_negotiation.cc:407] Connection header received
0809 04:51:08.438350 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:08.438356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:08.438435 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:08.438525 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:08.439572 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:08.440390 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:08.442285 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:08.442822 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:08.442939 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:08.444428 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:08.444436 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:08.444441 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:51:08.444460 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:08.444505 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:51:08.444521 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:08.444615 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:08.444831 (+ 216us) server_negotiation.cc:299] Negotiation successful
0809 04:51:08.444929 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:51:08.446425 15361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:08.435629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:34361 (local address 127.31.93.2:34223)
0809 04:51:08.436469 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:08.436503 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:08.438063 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:08.438760 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:08.438768 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:08.438858 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:08.439390 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:08.439406 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:08.440547 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:08.440553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:08.442147 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:08.442160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:08.443059 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:08.443065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:08.443351 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:08.444072 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:51:08.444076 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:08.444124 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:51:08.444228 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:51:08.444614 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:51:08.444623 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:51:08.444631 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 04:51:08.444808 (+ 177us) client_negotiation.cc:240] Negotiation successful
0809 04:51:08.445164 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":242,"threads_started":1}
I20260809 04:51:08.447762 15355 heartbeater.cc:347] Connected to a master server at 127.31.93.62:34361
I20260809 04:51:08.448156 15355 heartbeater.cc:464] Registering TS with master...
I20260809 04:51:08.448966 15355 heartbeater.cc:511] Master 127.31.93.62:34361 requested a full tablet report, sending...
I20260809 04:51:08.450493 14816 ts_manager.cc:194] Registered new tserver with Master: 92bb3ac16bd147c09ff0a5c7fc832b8c (127.31.93.2:44835)
I20260809 04:51:08.451546 14816 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.93.2:34223
W20260809 04:51:08.764034 15363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:51:08.764312 15363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:51:08.764426 15363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:51:08.771421 15363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:51:08.771528 15363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:51:08.771667 15363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:51:08.780791 15363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:51:08.782081 15363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:51:08.784001 15363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:51:08.796196 15372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:08.796162 15373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:08.796628 15377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:51:08.797091 15363 server_base.cc:1034] running on GCE node
I20260809 04:51:08.797688 15363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:51:08.798570 15363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:51:08.799783 15363 hybrid_clock.cc:648] HybridClock initialized: now 1786251068799730 us; error 45 us; skew 500 ppm
I20260809 04:51:08.800064 15363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:51:08.802170 15363 webserver.cc:460] Webserver started at http://127.31.93.3:38797/ using document root <none> and password file <none>
I20260809 04:51:08.802863 15363 fs_manager.cc:359] Metadata directory not provided
I20260809 04:51:08.802964 15363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:51:08.803265 15363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:51:08.806003 15363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "ef971890016e46aa81476d20176b314a"
format_stamp: "Formatted at 2026-08-09 04:51:08 on dist-test-slave-4tdj"
server_key: "6e594dda6c9b8108cdc113f76e540d6d"
server_key_iv: "4e8de8f5167a09137ed8274113c4987e"
server_key_version: "encryptionkey@0"
I20260809 04:51:08.806860 15363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "ef971890016e46aa81476d20176b314a"
format_stamp: "Formatted at 2026-08-09 04:51:08 on dist-test-slave-4tdj"
server_key: "6e594dda6c9b8108cdc113f76e540d6d"
server_key_iv: "4e8de8f5167a09137ed8274113c4987e"
server_key_version: "encryptionkey@0"
I20260809 04:51:08.811506 15363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260809 04:51:08.814635 15385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:08.815831 15363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260809 04:51:08.815979 15363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "ef971890016e46aa81476d20176b314a"
format_stamp: "Formatted at 2026-08-09 04:51:08 on dist-test-slave-4tdj"
server_key: "6e594dda6c9b8108cdc113f76e540d6d"
server_key_iv: "4e8de8f5167a09137ed8274113c4987e"
server_key_version: "encryptionkey@0"
I20260809 04:51:08.816167 15363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:51:08.845058 15363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:51:08.845938 15363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:51:08.846199 15363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:51:08.847199 15363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:51:08.848695 15363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:51:08.848793 15363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:08.848898 15363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:51:08.848971 15363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:08.878494 15363 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:43403
I20260809 04:51:08.878558 15579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:43403 every 8 connection(s)
I20260809 04:51:08.879755 15363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:51:08.888221 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 15363
I20260809 04:51:08.888368 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:51:08.888793 32116 external_mini_cluster.cc:1428] Setting key 447367f046b1ab22e7eb39dd447e2747
I20260809 04:51:08.891793 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:08.882240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:47959 (local address 127.31.93.62:34361)
0809 04:51:08.882438 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:08.882443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:08.883794 (+ 1351us) server_negotiation.cc:407] Connection header received
0809 04:51:08.885563 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:08.885568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:08.885637 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:08.885721 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:08.886705 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:08.887421 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:08.888931 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:08.889386 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:08.889518 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:08.891059 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:08.891064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:08.891067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:51:08.891083 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:08.891123 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:51:08.891134 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:08.891201 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:08.891437 (+ 236us) server_negotiation.cc:299] Negotiation successful
0809 04:51:08.891529 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 04:51:08.892748 15584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:08.882783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:34361 (local address 127.31.93.3:47959)
0809 04:51:08.883662 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:08.883698 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:08.885324 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:08.885899 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:08.885906 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:08.885988 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:08.886570 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:08.886588 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:08.887539 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:08.887543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:08.888792 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:08.888804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:08.889797 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:08.889804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:08.890083 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:08.890750 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:51:08.890753 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:08.890807 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:51:08.890899 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:51:08.891225 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:51:08.891236 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:51:08.891245 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 04:51:08.891416 (+ 171us) client_negotiation.cc:240] Negotiation successful
0809 04:51:08.891772 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":267,"threads_started":1}
I20260809 04:51:08.893958 15580 heartbeater.cc:347] Connected to a master server at 127.31.93.62:34361
I20260809 04:51:08.894225 15580 heartbeater.cc:464] Registering TS with master...
I20260809 04:51:08.894932 15580 heartbeater.cc:511] Master 127.31.93.62:34361 requested a full tablet report, sending...
I20260809 04:51:08.896221 14816 ts_manager.cc:194] Registered new tserver with Master: ef971890016e46aa81476d20176b314a (127.31.93.3:43403)
I20260809 04:51:08.896944 14816 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.93.3:47959
I20260809 04:51:08.903673 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 04:51:08.909385 15146 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:08.908093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.31.93.62:34361)
0809 04:51:08.908266 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:08.908269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:08.908671 (+ 402us) server_negotiation.cc:407] Connection header received
0809 04:51:08.908794 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:08.908797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:08.908861 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:08.908938 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:08.909219 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47106: BlockingRecv error: recv got EOF from 127.0.0.1:47106 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:51:08.910223 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 14933
I20260809 04:51:08.917018 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:08.911047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51648 (local address 127.31.93.1:43385)
0809 04:51:08.911533 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:08.911539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:08.911562 (+ 23us) server_negotiation.cc:407] Connection header received
0809 04:51:08.911626 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:08.911631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:08.911805 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:08.911919 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:08.912237 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:08.913166 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:08.914350 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:08.914875 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:08.914964 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:08.915798 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:08.915806 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:08.915816 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:51:08.916337 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:08.916383 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:51:08.916402 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:08.916481 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:08.916570 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 04:51:08.916690 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":241,"threads_started":1}
I20260809 04:51:08.918068 15134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:51:08.997489 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 14933
I20260809 04:51:09.020464 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 15151
I20260809 04:51:09.029625 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:09.021470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.31.93.2:44835)
0809 04:51:09.022226 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:09.022235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:09.022271 (+ 36us) server_negotiation.cc:407] Connection header received
0809 04:51:09.022381 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:09.022390 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:09.022716 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:09.022883 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:09.023419 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:09.024842 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:09.026212 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:09.027084 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:09.027195 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:09.028234 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:09.028242 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:09.028256 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:51:09.028820 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:09.028866 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:51:09.028886 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:09.029016 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:09.029147 (+ 131us) server_negotiation.cc:299] Negotiation successful
0809 04:51:09.029277 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":356,"threads_started":1}
I20260809 04:51:09.030874 15349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:51:09.110484 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 15151
I20260809 04:51:09.130487 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 15363
I20260809 04:51:09.138264 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:09.131466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.31.93.3:43403)
0809 04:51:09.132100 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:09.132107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:09.132136 (+ 29us) server_negotiation.cc:407] Connection header received
0809 04:51:09.132211 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:09.132218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:09.132429 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:09.132554 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:09.132965 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:09.134001 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:09.135318 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:09.135866 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:09.135962 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:09.136910 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:09.136918 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:09.136931 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:51:09.137503 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:09.137547 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:51:09.137578 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:09.137661 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:09.137756 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 04:51:09.137871 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":333,"threads_started":1}
I20260809 04:51:09.139498 15570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:51:09.219062 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 15363
I20260809 04:51:09.238760 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 14757
I20260809 04:51:09.239955 14873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:51:09.340546 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 14757
2026-08-09T04:51:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2390 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-09T04:51:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:51:09Z Disabled control of system clock
I20260809 04:51:09.417188 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:42035
--webserver_interface=127.31.93.62
--webserver_port=0
--builtin_ntp_servers=127.31.93.20:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:42035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 04:51:09.733469 15607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:51:09.733747 15607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:51:09.733867 15607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:51:09.740927 15607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:51:09.741016 15607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:51:09.741075 15607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:51:09.741151 15607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:51:09.741211 15607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:51:09.749985 15607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:42035
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:42035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:51:09.751199 15607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:51:09.752920 15607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:51:09.763696 15618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:09.763898 15615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:09.763659 15620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:51:09.764535 15607 server_base.cc:1034] running on GCE node
I20260809 04:51:09.765205 15607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:51:09.766520 15607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:51:09.767715 15607 hybrid_clock.cc:648] HybridClock initialized: now 1786251069767677 us; error 36 us; skew 500 ppm
I20260809 04:51:09.768021 15607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:51:09.773293 15607 webserver.cc:460] Webserver started at http://127.31.93.62:43605/ using document root <none> and password file <none>
I20260809 04:51:09.773926 15607 fs_manager.cc:359] Metadata directory not provided
I20260809 04:51:09.774020 15607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:51:09.774355 15607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:51:09.777143 15607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "51d674fd7ef34d4d8d4ffe639e18a0ba"
format_stamp: "Formatted at 2026-08-09 04:51:09 on dist-test-slave-4tdj"
server_key: "e6d59bc44510221263f68ce5cc64483d"
server_key_iv: "1c00df351e9c07bc085509c99b888dd1"
server_key_version: "encryptionkey@0"
I20260809 04:51:09.777892 15607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "51d674fd7ef34d4d8d4ffe639e18a0ba"
format_stamp: "Formatted at 2026-08-09 04:51:09 on dist-test-slave-4tdj"
server_key: "e6d59bc44510221263f68ce5cc64483d"
server_key_iv: "1c00df351e9c07bc085509c99b888dd1"
server_key_version: "encryptionkey@0"
I20260809 04:51:09.782686 15607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260809 04:51:09.785817 15628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:09.787067 15607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:51:09.787236 15607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "51d674fd7ef34d4d8d4ffe639e18a0ba"
format_stamp: "Formatted at 2026-08-09 04:51:09 on dist-test-slave-4tdj"
server_key: "e6d59bc44510221263f68ce5cc64483d"
server_key_iv: "1c00df351e9c07bc085509c99b888dd1"
server_key_version: "encryptionkey@0"
I20260809 04:51:09.787422 15607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:51:09.792649 15607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:51:09.793334 15607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:51:09.793592 15607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:51:09.810024 15607 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:42035
I20260809 04:51:09.810060 15709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:42035 every 8 connection(s)
I20260809 04:51:09.811208 15607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:51:09.814571 15710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:51:09.816339 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 15607
I20260809 04:51:09.816524 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:51:09.817162 32116 external_mini_cluster.cc:1428] Setting key ccffb1ee6f3a083849dca6cfe64e6217
I20260809 04:51:09.826149 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:09.819014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36746 (local address 127.31.93.62:42035)
0809 04:51:09.820119 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:09.820132 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:09.820181 (+ 49us) server_negotiation.cc:407] Connection header received
0809 04:51:09.822348 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:09.822374 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:09.822886 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:09.823298 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:09.824279 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:09.824287 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:09.824307 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:51:09.824324 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:09.824396 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:51:09.824416 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:09.824642 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:09.824790 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 04:51:09.825048 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":367,"threads_started":1}
I20260809 04:51:09.827080 15710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba: Bootstrap starting.
I20260809 04:51:09.830024 15710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba: Neither blocks nor log segments found. Creating new log.
I20260809 04:51:09.831084 15710 log.cc:824] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba: Log is configured to *not* fsync() on all Append() calls
I20260809 04:51:09.833634 15710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba: No bootstrap required, opened a new log
I20260809 04:51:09.840423 15710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d674fd7ef34d4d8d4ffe639e18a0ba" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 42035 } }
I20260809 04:51:09.840719 15710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:51:09.840816 15710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51d674fd7ef34d4d8d4ffe639e18a0ba, State: Initialized, Role: FOLLOWER
I20260809 04:51:09.841346 15710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d674fd7ef34d4d8d4ffe639e18a0ba" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 42035 } }
I20260809 04:51:09.841554 15710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:51:09.841679 15710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:51:09.841828 15710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:51:09.844046 15710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d674fd7ef34d4d8d4ffe639e18a0ba" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 42035 } }
I20260809 04:51:09.844507 15710 leader_election.cc:302] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51d674fd7ef34d4d8d4ffe639e18a0ba; no voters:
I20260809 04:51:09.845017 15710 leader_election.cc:288] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:51:09.845144 15716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:51:09.845801 15716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [term 1 LEADER]: Becoming Leader. State: Replica: 51d674fd7ef34d4d8d4ffe639e18a0ba, State: Running, Role: LEADER
I20260809 04:51:09.846366 15716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d674fd7ef34d4d8d4ffe639e18a0ba" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 42035 } }
I20260809 04:51:09.846891 15710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:51:09.849148 15719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51d674fd7ef34d4d8d4ffe639e18a0ba. Latest consensus state: current_term: 1 leader_uuid: "51d674fd7ef34d4d8d4ffe639e18a0ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d674fd7ef34d4d8d4ffe639e18a0ba" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 42035 } } }
I20260809 04:51:09.849560 15719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [sys.catalog]: This master's current role is: LEADER
I20260809 04:51:09.851490 15717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51d674fd7ef34d4d8d4ffe639e18a0ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d674fd7ef34d4d8d4ffe639e18a0ba" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 42035 } } }
I20260809 04:51:09.851852 15717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba [sys.catalog]: This master's current role is: LEADER
I20260809 04:51:09.852036 15727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:51:09.856294 15727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:51:09.863585 15727 catalog_manager.cc:1347] Generated new cluster ID: f80c38158c474e21b1d3ae10c1c13540
I20260809 04:51:09.863680 15727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:51:09.874956 15727 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:51:09.875310 15727 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:51:09.886790 15727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51d674fd7ef34d4d8d4ffe639e18a0ba: Generated new TSK 0
I20260809 04:51:09.887427 15727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:51:09.896548 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:42035
--builtin_ntp_servers=127.31.93.20:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 04:51:10.218736 15746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:51:10.218987 15746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:51:10.219121 15746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:51:10.226115 15746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:51:10.226202 15746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:51:10.226362 15746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:51:10.235314 15746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:42035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:51:10.236507 15746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:51:10.238222 15746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:51:10.250341 15754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:10.250456 15760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:10.250555 15756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:51:10.251641 15746 server_base.cc:1034] running on GCE node
I20260809 04:51:10.252195 15746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:51:10.253026 15746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:51:10.254230 15746 hybrid_clock.cc:648] HybridClock initialized: now 1786251070254186 us; error 41 us; skew 500 ppm
I20260809 04:51:10.254529 15746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:51:10.256536 15746 webserver.cc:460] Webserver started at http://127.31.93.1:45393/ using document root <none> and password file <none>
I20260809 04:51:10.257154 15746 fs_manager.cc:359] Metadata directory not provided
I20260809 04:51:10.257252 15746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:51:10.257543 15746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:51:10.260208 15746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "07d3eabe43af469ba66373a67706408a"
format_stamp: "Formatted at 2026-08-09 04:51:10 on dist-test-slave-4tdj"
server_key: "553b0327df2bc171693ab434f525041c"
server_key_iv: "6982cd6af185b4db036321060240cff6"
server_key_version: "encryptionkey@0"
I20260809 04:51:10.260943 15746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "07d3eabe43af469ba66373a67706408a"
format_stamp: "Formatted at 2026-08-09 04:51:10 on dist-test-slave-4tdj"
server_key: "553b0327df2bc171693ab434f525041c"
server_key_iv: "6982cd6af185b4db036321060240cff6"
server_key_version: "encryptionkey@0"
I20260809 04:51:10.265787 15746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 04:51:10.268965 15768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:10.270231 15746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 04:51:10.270394 15746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "07d3eabe43af469ba66373a67706408a"
format_stamp: "Formatted at 2026-08-09 04:51:10 on dist-test-slave-4tdj"
server_key: "553b0327df2bc171693ab434f525041c"
server_key_iv: "6982cd6af185b4db036321060240cff6"
server_key_version: "encryptionkey@0"
I20260809 04:51:10.270582 15746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:51:10.275743 15746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:51:10.276492 15746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:51:10.276734 15746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:51:10.277554 15746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:51:10.278924 15746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:51:10.279019 15746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:10.279130 15746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:51:10.279201 15746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:10.308419 15746 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:38323
I20260809 04:51:10.308468 15954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:38323 every 8 connection(s)
I20260809 04:51:10.309612 15746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:51:10.312614 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 15746
I20260809 04:51:10.312798 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:51:10.313406 32116 external_mini_cluster.cc:1428] Setting key 7f11290df501eb5b43109e1edf0f2e36
I20260809 04:51:10.316203 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:42035
--builtin_ntp_servers=127.31.93.20:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 04:51:10.317538 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:10.312017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:40223 (local address 127.31.93.62:42035)
0809 04:51:10.312202 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:10.312209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:10.313692 (+ 1483us) server_negotiation.cc:407] Connection header received
0809 04:51:10.315538 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:10.315543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:10.315609 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:10.315715 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:10.316943 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:10.316948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:10.316951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:51:10.316967 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:10.316999 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:51:10.317011 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:10.317102 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:10.317293 (+ 191us) server_negotiation.cc:299] Negotiation successful
0809 04:51:10.317338 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 04:51:10.318758 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:10.312515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:42035 (local address 127.31.93.1:40223)
0809 04:51:10.313534 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:10.313578 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:10.315283 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:10.315884 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:10.315892 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:10.315979 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:10.316617 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:51:10.316620 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:10.316666 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:51:10.316769 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:51:10.317104 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:51:10.317113 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:51:10.317120 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 04:51:10.317383 (+ 263us) client_negotiation.cc:240] Negotiation successful
0809 04:51:10.317630 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":322,"threads_started":1}
I20260809 04:51:10.320097 15958 heartbeater.cc:347] Connected to a master server at 127.31.93.62:42035
I20260809 04:51:10.320385 15958 heartbeater.cc:464] Registering TS with master...
I20260809 04:51:10.321007 15958 heartbeater.cc:511] Master 127.31.93.62:42035 requested a full tablet report, sending...
I20260809 04:51:10.323055 15648 ts_manager.cc:194] Registered new tserver with Master: 07d3eabe43af469ba66373a67706408a (127.31.93.1:38323)
W20260809 04:51:10.634213 15962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:51:10.634491 15962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:51:10.634622 15962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:51:10.641587 15962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:51:10.641683 15962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:51:10.641839 15962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:51:10.650935 15962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:42035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:51:10.652140 15962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:51:10.653898 15962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:51:10.666113 15971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:10.666118 15975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:10.666152 15973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:51:10.667111 15962 server_base.cc:1034] running on GCE node
I20260809 04:51:10.667749 15962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:51:10.668661 15962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:51:10.669879 15962 hybrid_clock.cc:648] HybridClock initialized: now 1786251070669833 us; error 50 us; skew 500 ppm
I20260809 04:51:10.670197 15962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:51:10.672443 15962 webserver.cc:460] Webserver started at http://127.31.93.2:46461/ using document root <none> and password file <none>
I20260809 04:51:10.673128 15962 fs_manager.cc:359] Metadata directory not provided
I20260809 04:51:10.673300 15962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:51:10.673637 15962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:51:10.679044 15962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "2f284dd58c5b4a96b21a3c7ff4d55626"
format_stamp: "Formatted at 2026-08-09 04:51:10 on dist-test-slave-4tdj"
server_key: "3bf90227354d639c6516a7b2fe04a4c9"
server_key_iv: "cd6277abc39c3f31c62e617959677cc1"
server_key_version: "encryptionkey@0"
I20260809 04:51:10.679803 15962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "2f284dd58c5b4a96b21a3c7ff4d55626"
format_stamp: "Formatted at 2026-08-09 04:51:10 on dist-test-slave-4tdj"
server_key: "3bf90227354d639c6516a7b2fe04a4c9"
server_key_iv: "cd6277abc39c3f31c62e617959677cc1"
server_key_version: "encryptionkey@0"
I20260809 04:51:10.684500 15962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 04:51:10.687718 15982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:10.688892 15962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 04:51:10.689038 15962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "2f284dd58c5b4a96b21a3c7ff4d55626"
format_stamp: "Formatted at 2026-08-09 04:51:10 on dist-test-slave-4tdj"
server_key: "3bf90227354d639c6516a7b2fe04a4c9"
server_key_iv: "cd6277abc39c3f31c62e617959677cc1"
server_key_version: "encryptionkey@0"
I20260809 04:51:10.689226 15962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:51:10.694442 15962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:51:10.695153 15962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:51:10.695385 15962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:51:10.696203 15962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:51:10.697463 15962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:51:10.697564 15962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:10.697669 15962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:51:10.697743 15962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:10.726714 15962 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:42141
I20260809 04:51:10.726774 16155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:42141 every 8 connection(s)
I20260809 04:51:10.727947 15962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:51:10.729867 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 15962
I20260809 04:51:10.730041 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:51:10.730542 32116 external_mini_cluster.cc:1428] Setting key 11d3280d1f6749b64f3c8d98d42e8ee3
I20260809 04:51:10.733373 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:42035
--builtin_ntp_servers=127.31.93.20:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 04:51:10.735982 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:10.730587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:51391 (local address 127.31.93.62:42035)
0809 04:51:10.730823 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:10.730829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:10.732296 (+ 1467us) server_negotiation.cc:407] Connection header received
0809 04:51:10.733987 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:10.733992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:10.734050 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:10.734144 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:10.735339 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:10.735344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:10.735347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:51:10.735364 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:10.735401 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:51:10.735414 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:10.735522 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:10.735656 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 04:51:10.735712 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 04:51:10.736778 16160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:10.731084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:42035 (local address 127.31.93.2:51391)
0809 04:51:10.732125 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:10.732188 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:10.733743 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:10.734354 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:10.734362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:10.734449 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:10.735055 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:51:10.735058 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:10.735102 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:51:10.735184 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:51:10.735498 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:51:10.735503 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:51:10.735508 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 04:51:10.735708 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 04:51:10.735890 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":293,"threads_started":1}
I20260809 04:51:10.737987 16157 heartbeater.cc:347] Connected to a master server at 127.31.93.62:42035
I20260809 04:51:10.738256 16157 heartbeater.cc:464] Registering TS with master...
I20260809 04:51:10.738811 16157 heartbeater.cc:511] Master 127.31.93.62:42035 requested a full tablet report, sending...
I20260809 04:51:10.740134 15648 ts_manager.cc:194] Registered new tserver with Master: 2f284dd58c5b4a96b21a3c7ff4d55626 (127.31.93.2:42141)
W20260809 04:51:11.065307 16161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:51:11.065579 16161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:51:11.065677 16161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:51:11.072664 16161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:51:11.072751 16161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:51:11.072890 16161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:51:11.081647 16161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:42035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:51:11.082875 16161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:51:11.084599 16161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:51:11.096998 16169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:11.096951 16170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:11.097031 16174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:51:11.097759 16161 server_base.cc:1034] running on GCE node
I20260809 04:51:11.098351 16161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:51:11.099223 16161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:51:11.100427 16161 hybrid_clock.cc:648] HybridClock initialized: now 1786251071100367 us; error 49 us; skew 500 ppm
I20260809 04:51:11.100721 16161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:51:11.102795 16161 webserver.cc:460] Webserver started at http://127.31.93.3:41687/ using document root <none> and password file <none>
I20260809 04:51:11.103434 16161 fs_manager.cc:359] Metadata directory not provided
I20260809 04:51:11.103538 16161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:51:11.103827 16161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:51:11.106577 16161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "3870e24fd47d45e89065119f13599534"
format_stamp: "Formatted at 2026-08-09 04:51:11 on dist-test-slave-4tdj"
server_key: "e816e3f527a514108681c81126c167f2"
server_key_iv: "dfaab2fb6f28efd707f759a168355c4d"
server_key_version: "encryptionkey@0"
I20260809 04:51:11.107342 16161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "3870e24fd47d45e89065119f13599534"
format_stamp: "Formatted at 2026-08-09 04:51:11 on dist-test-slave-4tdj"
server_key: "e816e3f527a514108681c81126c167f2"
server_key_iv: "dfaab2fb6f28efd707f759a168355c4d"
server_key_version: "encryptionkey@0"
I20260809 04:51:11.113489 16161 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260809 04:51:11.116804 16182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:11.118072 16161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:51:11.118227 16161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "3870e24fd47d45e89065119f13599534"
format_stamp: "Formatted at 2026-08-09 04:51:11 on dist-test-slave-4tdj"
server_key: "e816e3f527a514108681c81126c167f2"
server_key_iv: "dfaab2fb6f28efd707f759a168355c4d"
server_key_version: "encryptionkey@0"
I20260809 04:51:11.118486 16161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:51:11.123667 16161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:51:11.124403 16161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:51:11.124647 16161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:51:11.125509 16161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:51:11.126856 16161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:51:11.126955 16161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:11.127067 16161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:51:11.127135 16161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:11.156061 16161 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:35345
I20260809 04:51:11.156107 16351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:35345 every 8 connection(s)
I20260809 04:51:11.157310 16161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:51:11.157712 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 16161
I20260809 04:51:11.157882 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:51:11.158452 32116 external_mini_cluster.cc:1428] Setting key c23cc9df0d8f3e3aacabe23b0ceb4dd8
I20260809 04:51:11.165253 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:11.161008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:59403 (local address 127.31.93.62:42035)
0809 04:51:11.161225 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:11.161230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:11.161785 (+ 555us) server_negotiation.cc:407] Connection header received
0809 04:51:11.163437 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:11.163441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:11.163492 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:11.163574 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:11.164714 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:11.164718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:11.164721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:51:11.164734 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:11.164765 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:51:11.164776 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:11.164869 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:11.164996 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 04:51:11.165050 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 04:51:11.166124 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:11.160554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:42035 (local address 127.31.93.3:59403)
0809 04:51:11.161622 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:11.161668 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:11.163224 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:11.163761 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:11.163769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:11.163853 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:11.164455 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:51:11.164458 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:11.164495 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:51:11.164577 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:51:11.164849 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:51:11.164854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:51:11.164859 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 04:51:11.165057 (+ 198us) client_negotiation.cc:240] Negotiation successful
0809 04:51:11.165263 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":557,"threads_started":1}
I20260809 04:51:11.167294 16354 heartbeater.cc:347] Connected to a master server at 127.31.93.62:42035
I20260809 04:51:11.167565 16354 heartbeater.cc:464] Registering TS with master...
I20260809 04:51:11.168105 16354 heartbeater.cc:511] Master 127.31.93.62:42035 requested a full tablet report, sending...
I20260809 04:51:11.169350 15648 ts_manager.cc:194] Registered new tserver with Master: 3870e24fd47d45e89065119f13599534 (127.31.93.3:35345)
I20260809 04:51:11.173727 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 04:51:11.179761 15713 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:51:11.178382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.31.93.62:42035)
0809 04:51:11.178521 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:11.178525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:11.179008 (+ 483us) server_negotiation.cc:407] Connection header received
0809 04:51:11.179133 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:11.179136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:11.179184 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:11.179257 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:11.179518 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36758: BlockingRecv error: recv got EOF from 127.0.0.1:36758 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 04:51:11.181020 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 15746
I20260809 04:51:11.183991 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:11.181783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.31.93.1:38323)
0809 04:51:11.182250 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:11.182256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:11.182281 (+ 25us) server_negotiation.cc:407] Connection header received
0809 04:51:11.182370 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:11.182375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:11.182552 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:11.182667 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:11.183469 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:11.183475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:11.183485 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:51:11.183503 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:11.183539 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:51:11.183551 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:11.183654 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:11.183703 (+ 49us) server_negotiation.cc:299] Negotiation successful
0809 04:51:11.183777 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":216,"threads_started":1}
I20260809 04:51:11.185102 15949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:51:11.263949 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 15746
I20260809 04:51:11.285086 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 15962
I20260809 04:51:11.288386 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:11.285913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.31.93.2:42141)
0809 04:51:11.286488 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:11.286495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:11.286523 (+ 28us) server_negotiation.cc:407] Connection header received
0809 04:51:11.286584 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:11.286591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:11.286802 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:11.286926 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:11.287804 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:11.287810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:11.287822 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:51:11.287839 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:11.287878 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:51:11.287892 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:11.287991 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:11.288047 (+ 56us) server_negotiation.cc:299] Negotiation successful
0809 04:51:11.288138 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":281,"threads_started":1}
I20260809 04:51:11.289561 16150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:51:11.368242 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 15962
I20260809 04:51:11.388298 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 16161
I20260809 04:51:11.392071 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:11.389294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36358 (local address 127.31.93.3:35345)
0809 04:51:11.389951 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:11.389966 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:11.389997 (+ 31us) server_negotiation.cc:407] Connection header received
0809 04:51:11.390058 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:11.390064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:11.390340 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:11.390468 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:11.391424 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:11.391431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:11.391442 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:51:11.391461 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:11.391503 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:51:11.391518 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:11.391653 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:11.391704 (+ 51us) server_negotiation.cc:299] Negotiation successful
0809 04:51:11.391783 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":335,"threads_started":1}
I20260809 04:51:11.393280 16345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:51:11.471774 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 16161
I20260809 04:51:11.491840 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 15607
I20260809 04:51:11.493012 15703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:51:11.592092 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 15607
2026-08-09T04:51:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2252 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:51:11 dist-test-slave-4tdj krb5kdc[16386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:51:11 dist-test-slave-4tdj krb5kdc[16386](info): set up 2 sockets
Aug 09 04:51:11 dist-test-slave-4tdj krb5kdc[16386](info): commencing operation
krb5kdc: starting...
W20260809 04:51:12.795652 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:51:12 dist-test-slave-4tdj krb5kdc[16386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:51:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:51:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:51:13.018838 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:37231
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:37231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:51:13.343748 16404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:51:13.343995 16404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:51:13.344123 16404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:51:13.351168 16404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:51:13.351260 16404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:51:13.351334 16404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:51:13.351406 16404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:51:13.351465 16404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:51:13.360745 16404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:37231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:37231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:51:13.361910 16404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:51:13.363142 16404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:51:13.378561 16410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:13.378623 16414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:13.378659 16411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:51:13.379931 16404 server_base.cc:1034] running on GCE node
I20260809 04:51:13.380709 16404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:51:13.382000 16404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:51:13.383199 16404 hybrid_clock.cc:648] HybridClock initialized: now 1786251073383159 us; error 26 us; skew 500 ppm
I20260809 04:51:13.383525 16404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:51:13 dist-test-slave-4tdj krb5kdc[16386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251073, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:51:13.386411 16404 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:51:13.387779 16404 webserver.cc:460] Webserver started at http://127.31.93.62:44357/ using document root <none> and password file <none>
I20260809 04:51:13.388391 16404 fs_manager.cc:359] Metadata directory not provided
I20260809 04:51:13.388500 16404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:51:13.388795 16404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:51:13.391597 16404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "0a1d1ab24b944ba5b39cee23a9044836"
format_stamp: "Formatted at 2026-08-09 04:51:13 on dist-test-slave-4tdj"
server_key: "baa92c0c34dec3c451900a7a4e45685a"
server_key_iv: "1935e235531050764cc306da8c94bebe"
server_key_version: "encryptionkey@0"
I20260809 04:51:13.392302 16404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "0a1d1ab24b944ba5b39cee23a9044836"
format_stamp: "Formatted at 2026-08-09 04:51:13 on dist-test-slave-4tdj"
server_key: "baa92c0c34dec3c451900a7a4e45685a"
server_key_iv: "1935e235531050764cc306da8c94bebe"
server_key_version: "encryptionkey@0"
I20260809 04:51:13.397048 16404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:51:13.400183 16424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:13.401315 16404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:51:13.401458 16404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "0a1d1ab24b944ba5b39cee23a9044836"
format_stamp: "Formatted at 2026-08-09 04:51:13 on dist-test-slave-4tdj"
server_key: "baa92c0c34dec3c451900a7a4e45685a"
server_key_iv: "1935e235531050764cc306da8c94bebe"
server_key_version: "encryptionkey@0"
I20260809 04:51:13.401623 16404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:51:13.436678 16404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:51:13.440323 16404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:51:13.440586 16404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:51:13.457056 16404 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:37231
I20260809 04:51:13.457093 16503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:37231 every 8 connection(s)
I20260809 04:51:13.458369 16404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:51:13.461899 16504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:51:13.462167 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 16404
I20260809 04:51:13.462399 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:51:13.463027 32116 external_mini_cluster.cc:1428] Setting key 908306261ef4e9ee7bba2050646f4270
I20260809 04:51:13.474785 16504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836: Bootstrap starting.
Aug 09 04:51:13 dist-test-slave-4tdj krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:51:13.478459 16504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836: Neither blocks nor log segments found. Creating new log.
I20260809 04:51:13.479471 16504 log.cc:824] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836: Log is configured to *not* fsync() on all Append() calls
I20260809 04:51:13.482404 16504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836: No bootstrap required, opened a new log
I20260809 04:51:13.485241 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:13.464947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.31.93.62:37231)
0809 04:51:13.465718 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:13.465729 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:13.465768 (+ 39us) server_negotiation.cc:407] Connection header received
0809 04:51:13.467540 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:13.467561 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:13.467920 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:13.468233 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:13.469993 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:13.471275 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:13.473024 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:13.473656 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:13.473782 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:13.478383 (+ 4601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:13.478391 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:13.478405 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:51:13.478440 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:13.481405 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:13.482139 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:13.482146 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:13.482151 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:13.482338 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:13.482749 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:13.482753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:13.482755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:13.483248 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:13.483474 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:13.483889 (+ 415us) server_negotiation.cc:299] Negotiation successful
0809 04:51:13.484206 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":328,"threads_started":1}
I20260809 04:51:13.490573 16504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1d1ab24b944ba5b39cee23a9044836" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37231 } }
I20260809 04:51:13.490900 16504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:51:13.490990 16504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a1d1ab24b944ba5b39cee23a9044836, State: Initialized, Role: FOLLOWER
I20260809 04:51:13.491514 16504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1d1ab24b944ba5b39cee23a9044836" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37231 } }
I20260809 04:51:13.491744 16504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:51:13.491915 16504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:51:13.492100 16504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:51:13.494072 16504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1d1ab24b944ba5b39cee23a9044836" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37231 } }
I20260809 04:51:13.494544 16504 leader_election.cc:302] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a1d1ab24b944ba5b39cee23a9044836; no voters:
I20260809 04:51:13.495070 16504 leader_election.cc:288] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:51:13.495189 16514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:51:13.496122 16514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [term 1 LEADER]: Becoming Leader. State: Replica: 0a1d1ab24b944ba5b39cee23a9044836, State: Running, Role: LEADER
I20260809 04:51:13.496718 16514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1d1ab24b944ba5b39cee23a9044836" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37231 } }
I20260809 04:51:13.496932 16504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:51:13.499362 16516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a1d1ab24b944ba5b39cee23a9044836. Latest consensus state: current_term: 1 leader_uuid: "0a1d1ab24b944ba5b39cee23a9044836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1d1ab24b944ba5b39cee23a9044836" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37231 } } }
I20260809 04:51:13.499410 16515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a1d1ab24b944ba5b39cee23a9044836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1d1ab24b944ba5b39cee23a9044836" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37231 } } }
I20260809 04:51:13.499794 16516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [sys.catalog]: This master's current role is: LEADER
I20260809 04:51:13.499809 16515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836 [sys.catalog]: This master's current role is: LEADER
I20260809 04:51:13.501662 16520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:51:13.506656 16520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:51:13.515446 16520 catalog_manager.cc:1347] Generated new cluster ID: a7ba4924edc34dbf869126723622316b
I20260809 04:51:13.515560 16520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:51:13.531850 16520 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:51:13.533008 16520 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:51:13.541401 16520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836: Generated new TSK 0
I20260809 04:51:13.542098 16520 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:51:13.611677 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:37231
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:51:13.954221 16547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:51:13.954488 16547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:51:13.954591 16547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:51:13.961320 16547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:51:13.961409 16547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:51:13.961529 16547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:51:13.970731 16547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:37231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:51:13.971886 16547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:51:13.973130 16547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:51:13.986629 16554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:13.987075 16552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:13.987106 16558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:51:13.987555 16547 server_base.cc:1034] running on GCE node
I20260809 04:51:13.988091 16547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:51:13.988883 16547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:51:13.990070 16547 hybrid_clock.cc:648] HybridClock initialized: now 1786251073990033 us; error 27 us; skew 500 ppm
I20260809 04:51:13.990350 16547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:51:13 dist-test-slave-4tdj krb5kdc[16386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251073, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:51:13.993184 16547 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:51:13.994515 16547 webserver.cc:460] Webserver started at http://127.31.93.1:38611/ using document root <none> and password file <none>
I20260809 04:51:13.995141 16547 fs_manager.cc:359] Metadata directory not provided
I20260809 04:51:13.995234 16547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:51:13.995468 16547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:51:13.998193 16547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "236f9e45ed36479fa5f033280dd03d8c"
format_stamp: "Formatted at 2026-08-09 04:51:13 on dist-test-slave-4tdj"
server_key: "80ffa2c68ee49825f0608b54426f62c8"
server_key_iv: "b7b000bdde2c584b08df9d628ced1118"
server_key_version: "encryptionkey@0"
I20260809 04:51:13.999007 16547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "236f9e45ed36479fa5f033280dd03d8c"
format_stamp: "Formatted at 2026-08-09 04:51:13 on dist-test-slave-4tdj"
server_key: "80ffa2c68ee49825f0608b54426f62c8"
server_key_iv: "b7b000bdde2c584b08df9d628ced1118"
server_key_version: "encryptionkey@0"
I20260809 04:51:14.003808 16547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:51:14.007068 16571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:14.008278 16547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260809 04:51:14.008410 16547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "236f9e45ed36479fa5f033280dd03d8c"
format_stamp: "Formatted at 2026-08-09 04:51:13 on dist-test-slave-4tdj"
server_key: "80ffa2c68ee49825f0608b54426f62c8"
server_key_iv: "b7b000bdde2c584b08df9d628ced1118"
server_key_version: "encryptionkey@0"
I20260809 04:51:14.008566 16547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:51:14.020426 16547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:51:14.023833 16547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:51:14.024065 16547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:51:14.024843 16547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:51:14.026199 16547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:51:14.026310 16547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:14.026433 16547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:51:14.026495 16547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:14.055022 16547 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:42751
I20260809 04:51:14.055083 16728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:42751 every 8 connection(s)
I20260809 04:51:14.056183 16547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:51:14.066083 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 16547
I20260809 04:51:14.066280 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:51:14.066932 32116 external_mini_cluster.cc:1428] Setting key aad588eca4ceb20fda4aa17e684548e2
Aug 09 04:51:14 dist-test-slave-4tdj krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251073, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:51:14.074441 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:14.058467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:60537 (local address 127.31.93.62:37231)
0809 04:51:14.058879 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:14.058883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:14.059699 (+ 816us) server_negotiation.cc:407] Connection header received
0809 04:51:14.061428 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:14.061434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:14.061510 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:14.061605 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:14.063034 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:14.063751 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:14.065238 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:14.065627 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:14.065744 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:14.069231 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:14.069236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:14.069239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:51:14.069269 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:14.071650 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:14.072409 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:14.072412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:14.072414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:14.072526 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:14.072925 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:14.072928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:14.072930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:14.073174 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:14.073286 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:14.073975 (+ 689us) server_negotiation.cc:299] Negotiation successful
0809 04:51:14.074138 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":220,"threads_started":1}
I20260809 04:51:14.075603 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:14.058883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:37231 (local address 127.31.93.1:60537)
0809 04:51:14.059546 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:14.059591 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:14.061171 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:14.061837 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:14.061845 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:14.062344 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:14.062874 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:14.062890 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:14.063894 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:14.063898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:14.065086 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:14.065099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:14.065847 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:14.065854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:14.066060 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:14.066690 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:14.066711 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:14.069026 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:51:14.071790 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:51:14.071797 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:51:14.071810 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:51:14.072313 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:51:14.072611 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:51:14.072616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:51:14.072618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:51:14.072837 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:51:14.073281 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:51:14.073288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:51:14.073648 (+ 360us) client_negotiation.cc:769] Sending connection context
0809 04:51:14.073910 (+ 262us) client_negotiation.cc:240] Negotiation successful
0809 04:51:14.074400 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":255,"threads_started":1}
I20260809 04:51:14.077001 16729 heartbeater.cc:347] Connected to a master server at 127.31.93.62:37231
I20260809 04:51:14.077270 16729 heartbeater.cc:464] Registering TS with master...
I20260809 04:51:14.077941 16729 heartbeater.cc:511] Master 127.31.93.62:37231 requested a full tablet report, sending...
I20260809 04:51:14.080027 16457 ts_manager.cc:194] Registered new tserver with Master: 236f9e45ed36479fa5f033280dd03d8c (127.31.93.1:42751)
I20260809 04:51:14.082002 16457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:60537
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:51:14.118845 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:37231
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:51:14.430545 16746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:51:14.430785 16746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:51:14.430881 16746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:51:14.437703 16746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:51:14.437793 16746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:51:14.437898 16746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:51:14.446935 16746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:37231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:51:14.447988 16746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:51:14.449177 16746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:51:14.461508 16757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:14.461535 16755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:14.462031 16753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:51:14.463111 16746 server_base.cc:1034] running on GCE node
I20260809 04:51:14.463696 16746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:51:14.464612 16746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:51:14.465819 16746 hybrid_clock.cc:648] HybridClock initialized: now 1786251074465778 us; error 36 us; skew 500 ppm
I20260809 04:51:14.466104 16746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:51:14 dist-test-slave-4tdj krb5kdc[16386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251074, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:51:14.469017 16746 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:51:14.470427 16746 webserver.cc:460] Webserver started at http://127.31.93.2:35159/ using document root <none> and password file <none>
I20260809 04:51:14.471065 16746 fs_manager.cc:359] Metadata directory not provided
I20260809 04:51:14.471171 16746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:51:14.471444 16746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:51:14.474574 16746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "8e7af9b6800a4802a6f925ad2c018e33"
format_stamp: "Formatted at 2026-08-09 04:51:14 on dist-test-slave-4tdj"
server_key: "a1d1c8257c40e4474a0d1988c54d769c"
server_key_iv: "04faa4600a4e0dfd29ffed390cec424f"
server_key_version: "encryptionkey@0"
I20260809 04:51:14.475507 16746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "8e7af9b6800a4802a6f925ad2c018e33"
format_stamp: "Formatted at 2026-08-09 04:51:14 on dist-test-slave-4tdj"
server_key: "a1d1c8257c40e4474a0d1988c54d769c"
server_key_iv: "04faa4600a4e0dfd29ffed390cec424f"
server_key_version: "encryptionkey@0"
I20260809 04:51:14.482700 16746 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260809 04:51:14.486145 16766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:14.487399 16746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:51:14.487545 16746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "8e7af9b6800a4802a6f925ad2c018e33"
format_stamp: "Formatted at 2026-08-09 04:51:14 on dist-test-slave-4tdj"
server_key: "a1d1c8257c40e4474a0d1988c54d769c"
server_key_iv: "04faa4600a4e0dfd29ffed390cec424f"
server_key_version: "encryptionkey@0"
I20260809 04:51:14.487720 16746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:51:14.515630 16746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:51:14.519106 16746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:51:14.519357 16746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:51:14.520198 16746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:51:14.521425 16746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:51:14.521495 16746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:14.521616 16746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:51:14.521674 16746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:14.550778 16746 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:43699
I20260809 04:51:14.550824 16941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:43699 every 8 connection(s)
I20260809 04:51:14.551954 16746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:51:14.558147 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 16746
I20260809 04:51:14.558336 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:51:14.558761 32116 external_mini_cluster.cc:1428] Setting key 8bfbe20f566ace6d602733a2ef675cb6
Aug 09 04:51:14 dist-test-slave-4tdj krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251074, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:51:14.571167 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:14.554271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:58551 (local address 127.31.93.62:37231)
0809 04:51:14.554499 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:14.554506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:14.555645 (+ 1139us) server_negotiation.cc:407] Connection header received
0809 04:51:14.557323 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:14.557327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:14.557396 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:14.557480 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:14.559033 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:14.559884 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:14.561473 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:14.562064 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:14.562200 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:14.565471 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:14.565477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:14.565480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:51:14.565517 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:14.568346 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:14.569016 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:14.569021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:14.569024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:14.569177 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:14.569529 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:14.569533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:14.569535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:14.569837 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:14.569977 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:14.570601 (+ 624us) server_negotiation.cc:299] Negotiation successful
0809 04:51:14.570837 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 04:51:14.571996 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:14.554787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:37231 (local address 127.31.93.2:58551)
0809 04:51:14.555481 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:14.555528 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:14.557087 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:14.557710 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:14.557722 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:14.558233 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:14.558864 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:14.558884 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:14.560027 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:14.560031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:14.561325 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:14.561338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:14.562259 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:14.562264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:14.562490 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:14.563072 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:14.563094 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:14.565255 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:51:14.568478 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:51:14.568485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:51:14.568496 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:51:14.568895 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:51:14.569261 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:51:14.569264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:51:14.569266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:51:14.569435 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:51:14.569932 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:51:14.569942 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:51:14.570221 (+ 279us) client_negotiation.cc:769] Sending connection context
0809 04:51:14.570474 (+ 253us) client_negotiation.cc:240] Negotiation successful
0809 04:51:14.570895 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":282,"threads_started":1}
I20260809 04:51:14.573381 16943 heartbeater.cc:347] Connected to a master server at 127.31.93.62:37231
I20260809 04:51:14.573681 16943 heartbeater.cc:464] Registering TS with master...
I20260809 04:51:14.574491 16943 heartbeater.cc:511] Master 127.31.93.62:37231 requested a full tablet report, sending...
I20260809 04:51:14.576025 16457 ts_manager.cc:194] Registered new tserver with Master: 8e7af9b6800a4802a6f925ad2c018e33 (127.31.93.2:43699)
I20260809 04:51:14.576804 16457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:58551
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:51:14.613051 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:37231
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:51:14.925592 16949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:51:14.925841 16949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:51:14.925979 16949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:51:14.932765 16949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:51:14.932866 16949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:51:14.933025 16949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:51:14.941962 16949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:37231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:51:14.943066 16949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:51:14.944245 16949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:51:14.955938 16958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:14.956089 16962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:14.956116 16959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:51:14.956912 16949 server_base.cc:1034] running on GCE node
I20260809 04:51:14.957419 16949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:51:14.958201 16949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:51:14.959419 16949 hybrid_clock.cc:648] HybridClock initialized: now 1786251074959381 us; error 51 us; skew 500 ppm
I20260809 04:51:14.959715 16949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:51:14 dist-test-slave-4tdj krb5kdc[16386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251074, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:51:14.965234 16949 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:51:14.966599 16949 webserver.cc:460] Webserver started at http://127.31.93.3:45563/ using document root <none> and password file <none>
I20260809 04:51:14.967244 16949 fs_manager.cc:359] Metadata directory not provided
I20260809 04:51:14.967360 16949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:51:14.967633 16949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:51:14.970372 16949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "9a8a3db2ce6c4da39cfa2a83c10f1f58"
format_stamp: "Formatted at 2026-08-09 04:51:14 on dist-test-slave-4tdj"
server_key: "24f294da1179799bafc70dfa553d4311"
server_key_iv: "4daa3dfcbe068be6ef491062e2861d9f"
server_key_version: "encryptionkey@0"
I20260809 04:51:14.971101 16949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "9a8a3db2ce6c4da39cfa2a83c10f1f58"
format_stamp: "Formatted at 2026-08-09 04:51:14 on dist-test-slave-4tdj"
server_key: "24f294da1179799bafc70dfa553d4311"
server_key_iv: "4daa3dfcbe068be6ef491062e2861d9f"
server_key_version: "encryptionkey@0"
I20260809 04:51:14.976037 16949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 04:51:14.979154 16972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:14.980356 16949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:51:14.980504 16949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "9a8a3db2ce6c4da39cfa2a83c10f1f58"
format_stamp: "Formatted at 2026-08-09 04:51:14 on dist-test-slave-4tdj"
server_key: "24f294da1179799bafc70dfa553d4311"
server_key_iv: "4daa3dfcbe068be6ef491062e2861d9f"
server_key_version: "encryptionkey@0"
I20260809 04:51:14.980664 16949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:51:14.996956 16949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:51:15.000957 16949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:51:15.001235 16949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:51:15.002179 16949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:51:15.003547 16949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:51:15.003625 16949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:15.003746 16949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:51:15.003800 16949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:15.038779 16949 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:46809
I20260809 04:51:15.038820 17146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:46809 every 8 connection(s)
I20260809 04:51:15.039974 16949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:51:15.041529 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 16949
I20260809 04:51:15.041708 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:51:15.042249 32116 external_mini_cluster.cc:1428] Setting key 0ed8bef03b5353b185ed27d07f17693b
Aug 09 04:51:15 dist-test-slave-4tdj krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251074, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:51:15.059185 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.042423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:48785 (local address 127.31.93.62:37231)
0809 04:51:15.042593 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:15.042597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:15.043928 (+ 1331us) server_negotiation.cc:407] Connection header received
0809 04:51:15.046002 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:15.046007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:15.046089 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:15.046170 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:15.047612 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.048341 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.049804 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.050215 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.050355 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.053450 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:15.053454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:15.053458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:51:15.053489 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:15.056016 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:15.056843 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:15.056848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:15.056852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:15.057015 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:15.057468 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:15.057473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:15.057477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:15.057814 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:15.057944 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:15.058617 (+ 673us) server_negotiation.cc:299] Negotiation successful
0809 04:51:15.058834 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:51:15.060024 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.043031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:37231 (local address 127.31.93.3:48785)
0809 04:51:15.043786 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:15.043828 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:15.045758 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:15.046444 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:15.046452 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:15.046919 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:51:15.047451 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:15.047467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.048483 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.048487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:15.049670 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:15.049681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.050468 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.050476 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:15.050688 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.051231 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:51:15.051256 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:51:15.053260 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:51:15.056169 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:51:15.056177 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:51:15.056190 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:51:15.056707 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:51:15.057172 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:51:15.057177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:51:15.057181 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:51:15.057353 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:51:15.057935 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:51:15.057943 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:51:15.058238 (+ 295us) client_negotiation.cc:769] Sending connection context
0809 04:51:15.058501 (+ 263us) client_negotiation.cc:240] Negotiation successful
0809 04:51:15.058980 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":281,"threads_started":1}
I20260809 04:51:15.061154 17147 heartbeater.cc:347] Connected to a master server at 127.31.93.62:37231
I20260809 04:51:15.061417 17147 heartbeater.cc:464] Registering TS with master...
I20260809 04:51:15.062101 17147 heartbeater.cc:511] Master 127.31.93.62:37231 requested a full tablet report, sending...
I20260809 04:51:15.063474 16457 ts_manager.cc:194] Registered new tserver with Master: 9a8a3db2ce6c4da39cfa2a83c10f1f58 (127.31.93.3:46809)
I20260809 04:51:15.064240 16457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:48785
I20260809 04:51:15.069679 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:51:15.085253 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.074053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44482 (local address 127.31.93.62:37231)
0809 04:51:15.074255 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:15.074262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:15.074630 (+ 368us) server_negotiation.cc:407] Connection header received
0809 04:51:15.074752 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:15.074756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:15.074808 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:15.074879 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:15.076132 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.076804 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.077887 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.078283 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.078421 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.080460 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:15.080464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:15.080467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:51:15.080489 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:15.082692 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:15.083274 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:15.083280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:15.083284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:15.083414 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:15.083775 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:15.083780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:15.083782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:15.084155 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:15.084380 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:15.084741 (+ 361us) server_negotiation.cc:299] Negotiation successful
0809 04:51:15.084941 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 04:51:15.085254 16729 heartbeater.cc:503] Master 127.31.93.62:37231 was elected leader, sending a full tablet report...
I20260809 04:51:15.089150 16454 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:51:15.095904 16454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:51:15.117575 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.107302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:43699 (local address 127.0.0.1:51878)
0809 04:51:15.108470 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:15.108516 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:15.108707 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:15.109628 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:15.109635 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:15.109681 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:51:15.110118 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:15.110132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.112105 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.112110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:15.114563 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:15.114578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.116625 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.116631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:15.116839 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.116895 (+ 56us) client_negotiation.cc:769] Sending connection context
0809 04:51:15.117020 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 04:51:15.117205 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"threads_started":1}
I20260809 04:51:15.118467 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.107564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.31.93.2:43699)
0809 04:51:15.108132 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:15.108140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:15.108902 (+ 762us) server_negotiation.cc:407] Connection header received
0809 04:51:15.108994 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:15.109000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:15.109173 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:15.109362 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:51:15.110339 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.111942 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.114757 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.116188 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.116298 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.116481 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:15.117967 (+ 1486us) server_negotiation.cc:299] Negotiation successful
0809 04:51:15.118121 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":268,"threads_started":1}
I20260809 04:51:15.118460 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.108252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42128 (local address 127.31.93.1:42751)
0809 04:51:15.108841 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:15.108849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:15.110091 (+ 1242us) server_negotiation.cc:407] Connection header received
0809 04:51:15.110419 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:15.110428 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:15.110649 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:15.110816 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:51:15.111742 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.113254 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.115686 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.117066 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.117170 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.117380 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:15.117953 (+ 573us) server_negotiation.cc:299] Negotiation successful
0809 04:51:15.118122 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":2130,"threads_started":1}
I20260809 04:51:15.119180 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.107699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42751 (local address 127.0.0.1:42128)
0809 04:51:15.109960 (+ 2261us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:15.109988 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:15.110202 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:15.111066 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:15.111070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:15.111095 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:51:15.111520 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:15.111531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.113439 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.113444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:15.115506 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:15.115521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.117580 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.117586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:15.117759 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.117788 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 04:51:15.118718 (+ 930us) client_negotiation.cc:240] Negotiation successful
0809 04:51:15.118846 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2137,"thread_start_us":702,"threads_started":1}
I20260809 04:51:15.123472 16641 tablet_service.cc:1467] Processing CreateTablet for tablet 668983d3336d48c3ba984640f6928b73 (DEFAULT_TABLE table=test-table [id=9c31f75711c54a3cbd2c5fecf5a29b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:51:15.124972 16641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668983d3336d48c3ba984640f6928b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:51:15.125545 16837 tablet_service.cc:1467] Processing CreateTablet for tablet 668983d3336d48c3ba984640f6928b73 (DEFAULT_TABLE table=test-table [id=9c31f75711c54a3cbd2c5fecf5a29b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:51:15.127019 16837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668983d3336d48c3ba984640f6928b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:51:15.127991 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.109428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:46809 (local address 127.0.0.1:43228)
0809 04:51:15.118894 (+ 9466us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:15.118911 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:15.119014 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:15.123072 (+ 4058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:15.123085 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:15.123111 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:51:15.123261 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:15.123271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.124609 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.124614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:15.125897 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:15.125910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.127278 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.127284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:15.127448 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.127514 (+ 66us) client_negotiation.cc:769] Sending connection context
0809 04:51:15.127604 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 04:51:15.127734 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9385,"thread_start_us":1150,"threads_started":1}
I20260809 04:51:15.128118 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.110051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43228 (local address 127.31.93.3:46809)
0809 04:51:15.122541 (+ 12490us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:15.122549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:15.122591 (+ 42us) server_negotiation.cc:407] Connection header received
0809 04:51:15.122681 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:15.122687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:15.122908 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:15.123086 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:51:15.123415 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.124478 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.126072 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.127134 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.127254 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.127477 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:15.127648 (+ 171us) server_negotiation.cc:299] Negotiation successful
0809 04:51:15.127797 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12313,"thread_start_us":12131,"threads_started":1}
I20260809 04:51:15.130839 17046 tablet_service.cc:1467] Processing CreateTablet for tablet 668983d3336d48c3ba984640f6928b73 (DEFAULT_TABLE table=test-table [id=9c31f75711c54a3cbd2c5fecf5a29b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:51:15.132090 17046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668983d3336d48c3ba984640f6928b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:51:15.140120 17171 tablet_bootstrap.cc:492] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c: Bootstrap starting.
I20260809 04:51:15.142967 17173 tablet_bootstrap.cc:492] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33: Bootstrap starting.
I20260809 04:51:15.143872 17171 tablet_bootstrap.cc:654] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c: Neither blocks nor log segments found. Creating new log.
I20260809 04:51:15.144423 17175 tablet_bootstrap.cc:492] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58: Bootstrap starting.
I20260809 04:51:15.145164 17171 log.cc:824] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c: Log is configured to *not* fsync() on all Append() calls
I20260809 04:51:15.146946 17173 tablet_bootstrap.cc:654] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33: Neither blocks nor log segments found. Creating new log.
I20260809 04:51:15.147929 17175 tablet_bootstrap.cc:654] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58: Neither blocks nor log segments found. Creating new log.
I20260809 04:51:15.148343 17171 tablet_bootstrap.cc:492] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c: No bootstrap required, opened a new log
I20260809 04:51:15.148300 17173 log.cc:824] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33: Log is configured to *not* fsync() on all Append() calls
I20260809 04:51:15.148584 17171 ts_tablet_manager.cc:1397] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 04:51:15.149106 17175 log.cc:824] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58: Log is configured to *not* fsync() on all Append() calls
I20260809 04:51:15.151408 17173 tablet_bootstrap.cc:492] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33: No bootstrap required, opened a new log
I20260809 04:51:15.151696 17173 ts_tablet_manager.cc:1397] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 04:51:15.156241 17171 raft_consensus.cc:348] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7af9b6800a4802a6f925ad2c018e33" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43699 } } peers { permanent_uuid: "236f9e45ed36479fa5f033280dd03d8c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42751 } } peers { permanent_uuid: "9a8a3db2ce6c4da39cfa2a83c10f1f58" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46809 } }
I20260809 04:51:15.156567 17171 raft_consensus.cc:374] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:51:15.156697 17171 raft_consensus.cc:729] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 236f9e45ed36479fa5f033280dd03d8c, State: Initialized, Role: FOLLOWER
I20260809 04:51:15.156836 17175 tablet_bootstrap.cc:492] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58: No bootstrap required, opened a new log
I20260809 04:51:15.157090 17175 ts_tablet_manager.cc:1397] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260809 04:51:15.157333 17171 consensus_queue.cc:260] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7af9b6800a4802a6f925ad2c018e33" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43699 } } peers { permanent_uuid: "236f9e45ed36479fa5f033280dd03d8c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42751 } } peers { permanent_uuid: "9a8a3db2ce6c4da39cfa2a83c10f1f58" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46809 } }
I20260809 04:51:15.161460 17173 raft_consensus.cc:348] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7af9b6800a4802a6f925ad2c018e33" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43699 } } peers { permanent_uuid: "236f9e45ed36479fa5f033280dd03d8c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42751 } } peers { permanent_uuid: "9a8a3db2ce6c4da39cfa2a83c10f1f58" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46809 } }
I20260809 04:51:15.161847 17173 raft_consensus.cc:374] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:51:15.162003 17173 raft_consensus.cc:729] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7af9b6800a4802a6f925ad2c018e33, State: Initialized, Role: FOLLOWER
I20260809 04:51:15.162961 17171 ts_tablet_manager.cc:1428] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260809 04:51:15.163398 17173 consensus_queue.cc:260] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7af9b6800a4802a6f925ad2c018e33" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43699 } } peers { permanent_uuid: "236f9e45ed36479fa5f033280dd03d8c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42751 } } peers { permanent_uuid: "9a8a3db2ce6c4da39cfa2a83c10f1f58" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46809 } }
I20260809 04:51:15.164397 16943 heartbeater.cc:503] Master 127.31.93.62:37231 was elected leader, sending a full tablet report...
I20260809 04:51:15.165768 17173 ts_tablet_manager.cc:1428] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33: Time spent starting tablet: real 0.014s user 0.005s sys 0.008s
I20260809 04:51:15.166319 17179 raft_consensus.cc:482] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:51:15.166632 17179 raft_consensus.cc:504] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7af9b6800a4802a6f925ad2c018e33" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43699 } } peers { permanent_uuid: "236f9e45ed36479fa5f033280dd03d8c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42751 } } peers { permanent_uuid: "9a8a3db2ce6c4da39cfa2a83c10f1f58" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46809 } }
I20260809 04:51:15.166807 17175 raft_consensus.cc:348] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7af9b6800a4802a6f925ad2c018e33" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43699 } } peers { permanent_uuid: "236f9e45ed36479fa5f033280dd03d8c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42751 } } peers { permanent_uuid: "9a8a3db2ce6c4da39cfa2a83c10f1f58" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46809 } }
I20260809 04:51:15.167246 17175 raft_consensus.cc:374] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:51:15.167380 17175 raft_consensus.cc:729] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a8a3db2ce6c4da39cfa2a83c10f1f58, State: Initialized, Role: FOLLOWER
I20260809 04:51:15.167953 17175 consensus_queue.cc:260] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7af9b6800a4802a6f925ad2c018e33" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43699 } } peers { permanent_uuid: "236f9e45ed36479fa5f033280dd03d8c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42751 } } peers { permanent_uuid: "9a8a3db2ce6c4da39cfa2a83c10f1f58" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46809 } }
I20260809 04:51:15.168546 17179 leader_election.cc:288] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e7af9b6800a4802a6f925ad2c018e33 (127.31.93.2:43699), 9a8a3db2ce6c4da39cfa2a83c10f1f58 (127.31.93.3:46809)
I20260809 04:51:15.169318 17147 heartbeater.cc:503] Master 127.31.93.62:37231 was elected leader, sending a full tablet report...
I20260809 04:51:15.170531 17175 ts_tablet_manager.cc:1428] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260809 04:51:15.176812 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.170094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:36803 (local address 127.31.93.3:46809)
0809 04:51:15.170285 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:15.170337 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:15.171267 (+ 930us) server_negotiation.cc:407] Connection header received
0809 04:51:15.171547 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:15.171552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:15.171623 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:15.171681 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:51:15.172436 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.173449 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.175228 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.175964 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.176076 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.176171 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:15.176447 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 04:51:15.176570 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:51:15.176867 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.169139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:46809 (local address 127.31.93.1:36803)
0809 04:51:15.171164 (+ 2025us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:15.171184 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:15.171388 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:15.171868 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:15.171872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:15.171895 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:51:15.172275 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:15.172291 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.173608 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.173613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:15.175085 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:15.175098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.176164 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.176171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:15.176307 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.176330 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:51:15.176410 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 04:51:15.176534 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1887,"thread_start_us":1746,"threads_started":1}
I20260809 04:51:15.177198 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.169933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:51601 (local address 127.31.93.2:43699)
0809 04:51:15.170116 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:15.170122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:15.171608 (+ 1486us) server_negotiation.cc:407] Connection header received
0809 04:51:15.171904 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:15.171910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:15.171976 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:15.172061 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:51:15.173013 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.173918 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.175605 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.176352 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.176468 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.176555 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:15.176856 (+ 301us) server_negotiation.cc:299] Negotiation successful
0809 04:51:15.176974 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:51:15.177273 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.169742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:43699 (local address 127.31.93.1:51601)
0809 04:51:15.171528 (+ 1786us) negotiation.cc:107] Waiting for socket to connect
0809 04:51:15.171544 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:51:15.171705 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:51:15.172506 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:51:15.172511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:51:15.172537 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:51:15.172855 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:15.172867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.174047 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.174052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:15.175455 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:51:15.175468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.176564 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.176570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:51:15.176714 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.176737 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:51:15.176814 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 04:51:15.176937 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1648,"thread_start_us":1484,"threads_started":1}
I20260809 04:51:15.177837 17075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668983d3336d48c3ba984640f6928b73" candidate_uuid: "236f9e45ed36479fa5f033280dd03d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a8a3db2ce6c4da39cfa2a83c10f1f58" is_pre_election: true
I20260809 04:51:15.178224 17075 raft_consensus.cc:2454] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 236f9e45ed36479fa5f033280dd03d8c in term 0.
I20260809 04:51:15.178854 16576 leader_election.cc:302] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 236f9e45ed36479fa5f033280dd03d8c, 9a8a3db2ce6c4da39cfa2a83c10f1f58; no voters:
I20260809 04:51:15.178781 16863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668983d3336d48c3ba984640f6928b73" candidate_uuid: "236f9e45ed36479fa5f033280dd03d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7af9b6800a4802a6f925ad2c018e33" is_pre_election: true
I20260809 04:51:15.179169 17179 raft_consensus.cc:2781] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:51:15.179194 16863 raft_consensus.cc:2454] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 236f9e45ed36479fa5f033280dd03d8c in term 0.
I20260809 04:51:15.179319 17179 raft_consensus.cc:482] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:51:15.179423 17179 raft_consensus.cc:3037] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:51:15.181574 17179 raft_consensus.cc:504] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7af9b6800a4802a6f925ad2c018e33" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43699 } } peers { permanent_uuid: "236f9e45ed36479fa5f033280dd03d8c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42751 } } peers { permanent_uuid: "9a8a3db2ce6c4da39cfa2a83c10f1f58" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46809 } }
I20260809 04:51:15.182173 17179 leader_election.cc:288] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [CANDIDATE]: Term 1 election: Requested vote from peers 8e7af9b6800a4802a6f925ad2c018e33 (127.31.93.2:43699), 9a8a3db2ce6c4da39cfa2a83c10f1f58 (127.31.93.3:46809)
I20260809 04:51:15.182488 16863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668983d3336d48c3ba984640f6928b73" candidate_uuid: "236f9e45ed36479fa5f033280dd03d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7af9b6800a4802a6f925ad2c018e33"
I20260809 04:51:15.182682 16863 raft_consensus.cc:3037] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:51:15.182643 17075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668983d3336d48c3ba984640f6928b73" candidate_uuid: "236f9e45ed36479fa5f033280dd03d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a8a3db2ce6c4da39cfa2a83c10f1f58"
I20260809 04:51:15.182832 17075 raft_consensus.cc:3037] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:51:15.185794 17075 raft_consensus.cc:2454] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 236f9e45ed36479fa5f033280dd03d8c in term 1.
I20260809 04:51:15.185794 16863 raft_consensus.cc:2454] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 236f9e45ed36479fa5f033280dd03d8c in term 1.
I20260809 04:51:15.186177 16574 leader_election.cc:302] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 236f9e45ed36479fa5f033280dd03d8c, 8e7af9b6800a4802a6f925ad2c018e33; no voters:
I20260809 04:51:15.186460 17179 raft_consensus.cc:2781] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:51:15.186870 17179 raft_consensus.cc:686] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [term 1 LEADER]: Becoming Leader. State: Replica: 236f9e45ed36479fa5f033280dd03d8c, State: Running, Role: LEADER
I20260809 04:51:15.187310 17179 consensus_queue.cc:237] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7af9b6800a4802a6f925ad2c018e33" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43699 } } peers { permanent_uuid: "236f9e45ed36479fa5f033280dd03d8c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42751 } } peers { permanent_uuid: "9a8a3db2ce6c4da39cfa2a83c10f1f58" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46809 } }
I20260809 04:51:15.191177 16457 catalog_manager.cc:5495] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 236f9e45ed36479fa5f033280dd03d8c (127.31.93.1). New cstate: current_term: 1 leader_uuid: "236f9e45ed36479fa5f033280dd03d8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e7af9b6800a4802a6f925ad2c018e33" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "236f9e45ed36479fa5f033280dd03d8c" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a8a3db2ce6c4da39cfa2a83c10f1f58" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46809 } health_report { overall_health: UNKNOWN } } }
I20260809 04:51:15.217335 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.212856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42134 (local address 127.31.93.1:42751)
0809 04:51:15.213015 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:15.213019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:15.213061 (+ 42us) server_negotiation.cc:407] Connection header received
0809 04:51:15.213222 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:15.213226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:15.213299 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:15.213416 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:51:15.213702 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.214416 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.215636 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.216010 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.216097 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.216486 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:51:15.216940 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:51:15.217041 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:15.217090 (+ 49us) server_negotiation.cc:299] Negotiation successful
0809 04:51:15.217149 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 04:51:15.224820 16863 raft_consensus.cc:1261] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33 [term 1 FOLLOWER]: Refusing update from remote peer 236f9e45ed36479fa5f033280dd03d8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:51:15.224826 17075 raft_consensus.cc:1261] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58 [term 1 FOLLOWER]: Refusing update from remote peer 236f9e45ed36479fa5f033280dd03d8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:51:15.225739 17187 consensus_queue.cc:1035] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a8a3db2ce6c4da39cfa2a83c10f1f58" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:51:15.226033 17179 consensus_queue.cc:1035] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7af9b6800a4802a6f925ad2c018e33" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:51:15.250054 16457 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:51:15.250324 16457 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:51:15.253721 16457 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a1d1ab24b944ba5b39cee23a9044836: Sending DeleteTablet for 3 replicas of tablet 668983d3336d48c3ba984640f6928b73
I20260809 04:51:15.254804 16837 tablet_service.cc:1514] Processing DeleteTablet for tablet 668983d3336d48c3ba984640f6928b73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:51:15 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:51878
I20260809 04:51:15.255106 17046 tablet_service.cc:1514] Processing DeleteTablet for tablet 668983d3336d48c3ba984640f6928b73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:51:15 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:43228
I20260809 04:51:15.255323 16641 tablet_service.cc:1514] Processing DeleteTablet for tablet 668983d3336d48c3ba984640f6928b73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:51:15 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:42128
I20260809 04:51:15.257045 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 16547
I20260809 04:51:15.287644 17211 tablet_replica.cc:330] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33: stopping tablet replica
I20260809 04:51:15.288307 17216 tablet_replica.cc:330] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c: stopping tablet replica
I20260809 04:51:15.288619 17212 tablet_replica.cc:330] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58: stopping tablet replica
I20260809 04:51:15.288797 17216 raft_consensus.cc:2229] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:51:15.289060 17212 raft_consensus.cc:2229] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:51:15.289364 17216 raft_consensus.cc:2258] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:51:15.289585 17212 raft_consensus.cc:2258] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:51:15.290024 17211 raft_consensus.cc:2229] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:51:15.290571 17211 raft_consensus.cc:2258] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:51:15.292361 17212 ts_tablet_manager.cc:1905] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:51:15.293370 17216 ts_tablet_manager.cc:1905] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:51:15.294473 17211 ts_tablet_manager.cc:1905] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 04:51:15 dist-test-slave-4tdj krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:51:15.299366 17212 ts_tablet_manager.cc:1918] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:51:15.299610 17212 log.cc:1196] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/668983d3336d48c3ba984640f6928b73
I20260809 04:51:15.300222 17212 ts_tablet_manager.cc:1939] T 668983d3336d48c3ba984640f6928b73 P 9a8a3db2ce6c4da39cfa2a83c10f1f58: Deleting consensus metadata
I20260809 04:51:15.302124 16427 catalog_manager.cc:4841] TS 9a8a3db2ce6c4da39cfa2a83c10f1f58 (127.31.93.3:46809): tablet 668983d3336d48c3ba984640f6928b73 (table test-table [id=9c31f75711c54a3cbd2c5fecf5a29b9c]) successfully deleted
I20260809 04:51:15.302501 17211 ts_tablet_manager.cc:1918] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:51:15.302726 17211 log.cc:1196] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/668983d3336d48c3ba984640f6928b73
I20260809 04:51:15.303136 17211 ts_tablet_manager.cc:1939] T 668983d3336d48c3ba984640f6928b73 P 8e7af9b6800a4802a6f925ad2c018e33: Deleting consensus metadata
I20260809 04:51:15.303115 17216 ts_tablet_manager.cc:1918] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:51:15.303332 17216 log.cc:1196] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/668983d3336d48c3ba984640f6928b73
I20260809 04:51:15.303436 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.258098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42138 (local address 127.31.93.1:42751)
0809 04:51:15.258286 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:15.258321 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:15.280276 (+ 21955us) server_negotiation.cc:407] Connection header received
0809 04:51:15.280777 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:15.280784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:15.280874 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:15.280948 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:15.282850 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.284389 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.290003 (+ 5614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.290692 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.290804 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.296122 (+ 5318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:15.296140 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:15.296173 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:51:15.296233 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:15.299919 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:15.300772 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:15.300781 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:15.300785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:15.300977 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:15.301448 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:15.301454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:15.301457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:15.301877 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:15.302175 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:15.302764 (+ 589us) server_negotiation.cc:299] Negotiation successful
0809 04:51:15.302990 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 04:51:15.304373 16722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:51:15.304417 17216 ts_tablet_manager.cc:1939] T 668983d3336d48c3ba984640f6928b73 P 236f9e45ed36479fa5f033280dd03d8c: Deleting consensus metadata
I20260809 04:51:15.305024 16425 catalog_manager.cc:4841] TS 8e7af9b6800a4802a6f925ad2c018e33 (127.31.93.2:43699): tablet 668983d3336d48c3ba984640f6928b73 (table test-table [id=9c31f75711c54a3cbd2c5fecf5a29b9c]) successfully deleted
I20260809 04:51:15.400635 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 16547
I20260809 04:51:15.400719 16427 catalog_manager.cc:4841] TS 236f9e45ed36479fa5f033280dd03d8c (127.31.93.1:42751): tablet 668983d3336d48c3ba984640f6928b73 (table test-table [id=9c31f75711c54a3cbd2c5fecf5a29b9c]) successfully deleted
I20260809 04:51:15.423260 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 16746
Aug 09 04:51:15 dist-test-slave-4tdj krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:51:15.436898 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.423973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.31.93.2:43699)
0809 04:51:15.424167 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:15.424171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:15.424191 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:51:15.424326 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:15.424330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:15.424402 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:15.424482 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:15.425637 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.426532 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.427951 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.428323 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.428418 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.432222 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:15.432235 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:15.432250 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:51:15.432281 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:15.434734 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:15.435284 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:15.435291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:15.435293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:15.435404 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:15.435676 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:15.435679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:15.435681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:15.436026 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:15.436255 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:15.436424 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 04:51:15.436573 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 04:51:15.437497 16936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:51:15.523523 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 16746
I20260809 04:51:15.543865 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 16949
Aug 09 04:51:15 dist-test-slave-4tdj krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:51:15.557704 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:15.544610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.31.93.3:46809)
0809 04:51:15.544796 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:15.544801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:15.544822 (+ 21us) server_negotiation.cc:407] Connection header received
0809 04:51:15.544918 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:15.544922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:15.544988 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:15.545069 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:15.546273 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.547006 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.548166 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:15.548540 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:15.548626 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:15.552818 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:15.552834 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:15.552851 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:51:15.552887 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:15.555480 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:15.556033 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:15.556040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:15.556042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:15.556180 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:15.556462 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:15.556465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:15.556467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:15.556814 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:15.557022 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:15.557228 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 04:51:15.557389 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:51:15.558350 17141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:51:15.644279 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 16949
I20260809 04:51:15.667549 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 16404
I20260809 04:51:15.668672 16497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:51:15.774114 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 16404
2026-08-09T04:51:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4183 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:51:15 dist-test-slave-4tdj krb5kdc[17226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:51:15 dist-test-slave-4tdj krb5kdc[17226](info): set up 2 sockets
Aug 09 04:51:15 dist-test-slave-4tdj krb5kdc[17226](info): commencing operation
krb5kdc: starting...
W20260809 04:51:16.984401 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.003s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:51:17 dist-test-slave-4tdj krb5kdc[17226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:51:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:51:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.93.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.93.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.93.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/rangeradmin_127.31.93.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.93.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/rangeradmin_127.31.93.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.93.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.93.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.93.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/rangerlookup_127.31.93.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.93.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/rangerlookup_127.31.93.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.93.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.93.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.93.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/HTTP_127.31.93.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.93.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/HTTP_127.31.93.20@KRBTEST.COM.keytab.
I20260809 04:51:17.329305 32116 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAAy6Yk/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/postgres -l logfile start
2026-08-09 04:51:20.375 UTC [17261] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 04:51:20.375 UTC [17261] LOG: listening on IPv4 address "127.31.93.20", port 43277
2026-08-09 04:51:20.378 UTC [17261] LOG: listening on Unix socket "/tmp/.s.PGSQL.43277"
2026-08-09 04:51:20.412 UTC [17263] LOG: database system was shut down at 2026-08-09 04:51:19 UTC
2026-08-09 04:51:20.417 UTC [17261] LOG: database system is ready to accept connections
I20260809 04:51:21.807061 32116 mini_postgres.cc:96] Postgres bound to 43277
2026-08-09 04:51:21.828 UTC [17279] FATAL: database "slave" does not exist
127.31.93.20:43277 - accepting connections
I20260809 04:51:21.829101 32116 mini_ranger.cc:161] Starting Ranger...
I20260809 04:51:21.855180 32116 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 04:51:22.241326 32116 mini_ranger.cc:87] Created ranger Postgres database
I20260809 04:51:22.241446 32116 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-admin
2026-08-09 04:51:22,544 [I] DB FLAVOR :POSTGRES
2026-08-09 04:51:22,544 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 04:51:22,544 [I] Checking connection..
2026-08-09 04:51:22,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 04:51:22,775 [I] Checking connection passed.
2026-08-09 04:51:22,776 [I] --------- Verifying version history table ---------
2026-08-09 04:51:22,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 04:51:23,003 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 04:51:23,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 04:51:23,260 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 04:51:23,260 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 04:51:23,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 04:51:23.508 UTC [17383] WARNING: there is no transaction in progress
2026-08-09 04:51:23,514 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 04:51:23,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 04:51:23,764 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 04:51:23,765 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 04:51:23,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 04:51:24,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 04:51:24,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4tdj.c.gcp-upstream.internal','N') ;"
2026-08-09 04:51:24,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 04:51:24,774 [I] Table x_portal_user does not exist in database ranger
2026-08-09 04:51:24,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 04:51:25,030 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 04:51:25,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 04:51:25,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 04:51:25,510 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 04:51:25,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 04:51:26.351 UTC [17555] WARNING: there is no transaction in progress
2026-08-09 04:51:26.365 UTC [17555] WARNING: there is no transaction in progress
2026-08-09 04:51:26.380 UTC [17555] WARNING: there is no transaction in progress
2026-08-09 04:51:26.392 UTC [17555] WARNING: there is no transaction in progress
2026-08-09 04:51:26.406 UTC [17555] WARNING: there is no transaction in progress
2026-08-09 04:51:26.421 UTC [17555] WARNING: there is no transaction in progress
2026-08-09 04:51:26.468 UTC [17555] WARNING: there is no transaction in progress
2026-08-09 04:51:26.474 UTC [17555] WARNING: there is no transaction in progress
2026-08-09 04:51:26.487 UTC [17555] WARNING: there is no transaction in progress
2026-08-09 04:51:26.496 UTC [17555] WARNING: there is no transaction in progress
2026-08-09 04:51:26.505 UTC [17555] WARNING: there is no transaction in progress
2026-08-09 04:51:26.517 UTC [17555] WARNING: there is no transaction in progress
2026-08-09 04:51:26.525 UTC [17555] WARNING: there is no transaction in progress
2026-08-09 04:51:26.531 UTC [17555] WARNING: there is no transaction in progress
2026-08-09 04:51:26.536 UTC [17555] WARNING: there is no transaction in progress
2026-08-09 04:51:26.542 UTC [17555] WARNING: there is no transaction in progress
2026-08-09 04:51:27,073 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 04:51:27,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 04:51:27,303 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 04:51:27,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 04:51:27,539 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 04:51:27,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 04:51:27,788 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 04:51:27,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 04:51:28,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 04:51:28,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4tdj.c.gcp-upstream.internal';"
2026-08-09 04:51:28,497 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 04:51:28,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 04:51:28,742 [I] DB_PATCHES have already been applied
I20260809 04:51:28.748623 32116 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAAy6Yk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 04:51:28.748709 32116 mini_ranger.cc:193] Using host: 127.31.93.20
I20260809 04:51:28.767406 32116 mini_ranger.cc:227] Ranger admin URL: http://127.31.93.20:35687
Aug 09, 2026 4:51:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 4:51:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 4:51:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 4:51:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 4:51:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 4:51:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 4:51:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 04:51:30.018464 32116 mini_ranger.cc:160] Time spent starting Ranger: real 8.189s user 0.006s sys 0.045s
Aug 09, 2026 4:51:49 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 4:51:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 09, 2026 4:51:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 4:51:49 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 4:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 4:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 4:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 4:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 4:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 4:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 4:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 4:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 4:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 4:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 4:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 4:51:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 4:51:49 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 4:51:50 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 04:51:50.595602 32116 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.93.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.93.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.93.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/rangerkms_127.31.93.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.93.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/rangerkms_127.31.93.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.93.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.93.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.93.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/HTTP_127.31.93.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.93.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/HTTP_127.31.93.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 04:51:50 dist-test-slave-4tdj krb5kdc[17226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251110, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 04:51:50.763149 32116 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 04:51:50.783314 32116 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 04:51:51.436906 32116 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 04:51:51.437071 32116 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-kms
I20260809 04:51:51.437208 32116 mini_ranger_kms.cc:227] Using postgres at 127.31.93.20:43277
2026-08-09 04:51:51,950 [I] DB FLAVOR :POSTGRES
2026-08-09 04:51:51,950 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 04:51:51,950 [I] Checking connection
2026-08-09 04:51:51,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 04:51:52,202 [I] connection success
2026-08-09 04:51:52,203 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 04:51:52,203 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 04:51:52,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 04:51:52,448 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 04:51:52,449 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 04:51:52,449 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 04:51:52,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:43277/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 04:51:52,690 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 04:51:53.111416 32116 mini_ranger_kms.cc:310] Created kms service
I20260809 04:51:53.271960 32116 mini_ranger_kms.cc:326] Created kudu user
I20260809 04:51:53.312386 32116 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 04:51:53 dist-test-slave-4tdj krb5kdc[17226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251110, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:51:53.684598 32116 mini_ranger_kms.cc:384] Added ranger policy
I20260809 04:51:53.685009 32116 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 04:51:53.685073 32116 mini_ranger_kms.cc:240] Using host: 127.31.93.20
I20260809 04:51:53.709965 32116 mini_ranger_kms.cc:276] Ranger KMS PID: 17851
I20260809 04:51:53.710180 32116 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.93.20:56663
Aug 09, 2026 4:51:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 4:51:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 4:51:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 4:51:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 4:51:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 4:51:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 04:51:55 dist-test-slave-4tdj krb5kdc[17226](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786251115, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.93.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:51:56.225018 32116 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.462s user 0.138s sys 0.397s
I20260809 04:51:56.225378 32116 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 04:51:56.225608 32116 mini_ranger_kms.cc:401] 127.31.93.20:56663/kms/v1/keys
Aug 09 04:51:56 dist-test-slave-4tdj krb5kdc[17226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251115, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.93.20@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
Aug 09 04:51:56 dist-test-slave-4tdj krb5kdc[17226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251115, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.93.20@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
Aug 09 04:51:58 dist-test-slave-4tdj krb5kdc[17226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:51:59.067350 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:32805
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:42233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:32805
--ranger_config_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.93.20:56663/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:51:59.467975 17933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:51:59.468254 17933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:51:59.468390 17933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:51:59.476042 17933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:51:59.476161 17933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:51:59.476243 17933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:51:59.476316 17933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:51:59.476383 17933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:51:59.487893 17933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:42233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.93.20:56663/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:32805
--ranger_config_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:32805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:51:59.489326 17933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:51:59.490851 17933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:51:59.503823 17943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:59.503790 17945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:51:59.504207 17947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:51:59.504653 17933 server_base.cc:1034] running on GCE node
I20260809 04:51:59.505362 17933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:51:59.506673 17933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:51:59.507906 17933 hybrid_clock.cc:648] HybridClock initialized: now 1786251119507855 us; error 30 us; skew 500 ppm
I20260809 04:51:59.508239 17933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:51:59 dist-test-slave-4tdj krb5kdc[17226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251119, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:51:59.514966 17933 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:51:59.516409 17933 webserver.cc:460] Webserver started at http://127.31.93.62:44843/ using document root <none> and password file <none>
I20260809 04:51:59.517056 17933 fs_manager.cc:359] Metadata directory not provided
I20260809 04:51:59.517163 17933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:51:59.517515 17933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 04:51:59 dist-test-slave-4tdj krb5kdc[17226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251119, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:51:59.575009 17933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "137fcc7ead854c6088a4b6c3c61232cb"
format_stamp: "Formatted at 2026-08-09 04:51:59 on dist-test-slave-4tdj"
server_key: "f66ab6383a7aaebadb5fe66e484bb476"
server_key_iv: "bbbcaafece7d259b5fa49d065dd55dc1"
server_key_version: "kuduclusterkey@0"
I20260809 04:51:59.575824 17933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "137fcc7ead854c6088a4b6c3c61232cb"
format_stamp: "Formatted at 2026-08-09 04:51:59 on dist-test-slave-4tdj"
server_key: "f66ab6383a7aaebadb5fe66e484bb476"
server_key_iv: "bbbcaafece7d259b5fa49d065dd55dc1"
server_key_version: "kuduclusterkey@0"
I20260809 04:51:59.581566 17933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 04:51:59.597038 17961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:51:59.598484 17933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 04:51:59.598690 17933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "137fcc7ead854c6088a4b6c3c61232cb"
format_stamp: "Formatted at 2026-08-09 04:51:59 on dist-test-slave-4tdj"
server_key: "f66ab6383a7aaebadb5fe66e484bb476"
server_key_iv: "bbbcaafece7d259b5fa49d065dd55dc1"
server_key_version: "kuduclusterkey@0"
I20260809 04:51:59.598897 17933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:51:59.630448 17933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:51:59.634871 17933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:51:59.635236 17933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:51:59.656009 17933 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:32805
I20260809 04:51:59.656050 18048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:32805 every 8 connection(s)
I20260809 04:51:59.657326 17933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:51:59.661172 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 17933
I20260809 04:51:59.661366 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:51:59.661701 18049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:51:59.662061 32116 external_mini_cluster.cc:1428] Setting key dc409c1210508490f175cc4462619e5c
I20260809 04:51:59.676023 18049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb: Bootstrap starting.
Aug 09 04:51:59 dist-test-slave-4tdj krb5kdc[17226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:51:59.679597 18049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb: Neither blocks nor log segments found. Creating new log.
I20260809 04:51:59.681017 18049 log.cc:824] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb: Log is configured to *not* fsync() on all Append() calls
I20260809 04:51:59.684231 18049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb: No bootstrap required, opened a new log
I20260809 04:51:59.686429 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:51:59.664194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55338 (local address 127.31.93.62:32805)
0809 04:51:59.665063 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0809 04:51:59.665074 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:51:59.665117 (+ 43us) server_negotiation.cc:407] Connection header received
0809 04:51:59.667863 (+ 2746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:51:59.667886 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:51:59.668270 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:51:59.668586 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:51:59.670200 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:59.672310 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:59.673661 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:51:59.674996 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:51:59.675186 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:51:59.678704 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:51:59.678714 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:51:59.678731 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:51:59.678779 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:51:59.682246 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:59.682908 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:59.682916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:59.682923 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:59.683131 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:51:59.683443 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:51:59.683448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:51:59.683451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:51:59.684100 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:51:59.684382 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0809 04:51:59.684727 (+ 345us) server_negotiation.cc:299] Negotiation successful
0809 04:51:59.685132 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":436,"threads_started":1}
I20260809 04:51:59.693094 18049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "137fcc7ead854c6088a4b6c3c61232cb" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 32805 } }
I20260809 04:51:59.693521 18049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:51:59.693629 18049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137fcc7ead854c6088a4b6c3c61232cb, State: Initialized, Role: FOLLOWER
I20260809 04:51:59.694115 18049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "137fcc7ead854c6088a4b6c3c61232cb" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 32805 } }
I20260809 04:51:59.694424 18049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:51:59.694643 18049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:51:59.694890 18049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:51:59.697229 18049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "137fcc7ead854c6088a4b6c3c61232cb" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 32805 } }
I20260809 04:51:59.697641 18049 leader_election.cc:302] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 137fcc7ead854c6088a4b6c3c61232cb; no voters:
I20260809 04:51:59.698437 18049 leader_election.cc:288] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:51:59.698529 18054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:51:59.699287 18054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [term 1 LEADER]: Becoming Leader. State: Replica: 137fcc7ead854c6088a4b6c3c61232cb, State: Running, Role: LEADER
I20260809 04:51:59.699874 18054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "137fcc7ead854c6088a4b6c3c61232cb" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 32805 } }
I20260809 04:51:59.700448 18049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:51:59.702569 18056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 137fcc7ead854c6088a4b6c3c61232cb. Latest consensus state: current_term: 1 leader_uuid: "137fcc7ead854c6088a4b6c3c61232cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "137fcc7ead854c6088a4b6c3c61232cb" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 32805 } } }
I20260809 04:51:59.703152 18056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [sys.catalog]: This master's current role is: LEADER
I20260809 04:51:59.703013 18055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "137fcc7ead854c6088a4b6c3c61232cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "137fcc7ead854c6088a4b6c3c61232cb" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 32805 } } }
I20260809 04:51:59.703455 18055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb [sys.catalog]: This master's current role is: LEADER
I20260809 04:51:59.705157 18064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:51:59.711447 18064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:51:59.712160 18049 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 04:51:59.731594 18064 catalog_manager.cc:1347] Generated new cluster ID: 1080a896ccaa47a2b218e240d3a2eda8
I20260809 04:51:59.731747 18064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:51:59.783295 18064 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:51:59.785591 18064 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:51:59.801586 18064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb: Generated new TSK 0
I20260809 04:51:59.802759 18064 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:52:00.545672 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.93.20:56663/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:32805
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:42233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:52:00 dist-test-slave-4tdj krb5kdc[17226](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786251120, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 04:52:00.995666 18116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:52:00.996233 18116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:52:00.996420 18116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:52:01.010236 18116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:52:01.010493 18116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:52:01.010700 18116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:52:01.027885 18116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:42233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.93.20:56663/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:32805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:52:01.029986 18116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:52:01.031385 18116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:52:01.047293 18130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:01.047323 18131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:01.047600 18135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:52:01.047914 18116 server_base.cc:1034] running on GCE node
I20260809 04:52:01.048637 18116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:52:01.049582 18116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:52:01.050807 18116 hybrid_clock.cc:648] HybridClock initialized: now 1786251121050735 us; error 54 us; skew 500 ppm
I20260809 04:52:01.051131 18116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:52:01 dist-test-slave-4tdj krb5kdc[17226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251121, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:52:01.054826 18116 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:52:01.060935 18116 webserver.cc:460] Webserver started at http://127.31.93.1:43577/ using document root <none> and password file <none>
I20260809 04:52:01.061733 18116 fs_manager.cc:359] Metadata directory not provided
I20260809 04:52:01.061867 18116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:52:01.062260 18116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 04:52:01 dist-test-slave-4tdj krb5kdc[17226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251121, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:52:01.082515 18116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "4529e4ac5d7e422c9d030390fd1922ec"
format_stamp: "Formatted at 2026-08-09 04:52:01 on dist-test-slave-4tdj"
server_key: "0beca28b543828122903df1501affe27"
server_key_iv: "f0e2b9a10ce991a715c783464eaf8549"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:01.083664 18116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "4529e4ac5d7e422c9d030390fd1922ec"
format_stamp: "Formatted at 2026-08-09 04:52:01 on dist-test-slave-4tdj"
server_key: "0beca28b543828122903df1501affe27"
server_key_iv: "f0e2b9a10ce991a715c783464eaf8549"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:01.108740 18116 fs_manager.cc:691] Time spent creating directory manager: real 0.025s user 0.012s sys 0.012s
I20260809 04:52:01.135396 18145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:01.137490 18116 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.007s
I20260809 04:52:01.137741 18116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "4529e4ac5d7e422c9d030390fd1922ec"
format_stamp: "Formatted at 2026-08-09 04:52:01 on dist-test-slave-4tdj"
server_key: "0beca28b543828122903df1501affe27"
server_key_iv: "f0e2b9a10ce991a715c783464eaf8549"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:01.138063 18116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:52:01.195926 18116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:52:01.201411 18116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:52:01.201761 18116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:52:01.203200 18116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:52:01.205171 18116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:52:01.205293 18116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:01.205588 18116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:52:01.205780 18116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:52:01 dist-test-slave-4tdj krb5kdc[17226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251120, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:52:01.361080 18116 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:36161
I20260809 04:52:01.361137 18317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:36161 every 8 connection(s)
I20260809 04:52:01.362988 18116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:52:01.368048 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 18116
I20260809 04:52:01.368280 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:52:01.369269 32116 external_mini_cluster.cc:1428] Setting key 21c688a17e1202380329f53f2b85d40d
Aug 09 04:52:01 dist-test-slave-4tdj krb5kdc[17226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251121, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:52:01.407208 18323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:01.374494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:32805 (local address 127.31.93.1:51683)
0809 04:52:01.384409 (+ 9915us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:01.384463 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:01.386880 (+ 2417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:01.389510 (+ 2630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:01.389522 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:01.390130 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:01.390940 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:01.390964 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:01.392419 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:01.392424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:01.393776 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:01.393796 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:01.394824 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:01.394831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:01.395154 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:01.395909 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:01.395937 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:01.398593 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:01.402472 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:01.402480 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:01.402497 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:01.403098 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:01.403543 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:01.403548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:01.403552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:01.403799 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:01.404423 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:01.404430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:01.404829 (+ 399us) client_negotiation.cc:769] Sending connection context
0809 04:52:01.405095 (+ 266us) client_negotiation.cc:240] Negotiation successful
0809 04:52:01.405685 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9546,"thread_start_us":12735,"threads_started":1}
I20260809 04:52:01.408418 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:01.378662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:51683 (local address 127.31.93.62:32805)
0809 04:52:01.388799 (+ 10137us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:01.388804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:01.388823 (+ 19us) server_negotiation.cc:407] Connection header received
0809 04:52:01.388881 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:01.388886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:01.388968 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:01.389077 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:01.391154 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:01.392205 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:01.393928 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:01.394571 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:01.398788 (+ 4217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:01.399014 (+ 226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:01.399021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:01.399025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:01.399063 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:01.402325 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:01.403242 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:01.403247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:01.403250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:01.403399 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:01.403937 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:01.403942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:01.403945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:01.404285 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:01.407686 (+ 3401us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:01.407813 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 04:52:01.408040 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10025,"thread_start_us":9912,"threads_started":1}
I20260809 04:52:01.409817 18318 heartbeater.cc:347] Connected to a master server at 127.31.93.62:32805
I20260809 04:52:01.410136 18318 heartbeater.cc:464] Registering TS with master...
I20260809 04:52:01.411020 18318 heartbeater.cc:511] Master 127.31.93.62:32805 requested a full tablet report, sending...
I20260809 04:52:01.413481 17997 ts_manager.cc:194] Registered new tserver with Master: 4529e4ac5d7e422c9d030390fd1922ec (127.31.93.1:36161)
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
I20260809 04:52:01.415761 17997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:51683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:52:01.447042 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.93.20:56663/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:32805
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:42233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:52:01 dist-test-slave-4tdj krb5kdc[17226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251120, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
W20260809 04:52:01.913888 18329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:52:01.914142 18329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:52:01.914242 18329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:52:01.921393 18329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:52:01.921499 18329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:52:01.921623 18329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:52:01.931138 18329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:42233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.93.20:56663/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:32805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:52:01.932331 18329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:52:01.933583 18329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:52:01.946098 18356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:01.946154 18353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:01.946158 18354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:52:01.947284 18329 server_base.cc:1034] running on GCE node
I20260809 04:52:01.947844 18329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:52:01.948755 18329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:52:01.949965 18329 hybrid_clock.cc:648] HybridClock initialized: now 1786251121949896 us; error 46 us; skew 500 ppm
I20260809 04:52:01.950281 18329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:52:01 dist-test-slave-4tdj krb5kdc[17226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251121, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:52:01.953402 18329 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:52:01.954871 18329 webserver.cc:460] Webserver started at http://127.31.93.2:38109/ using document root <none> and password file <none>
I20260809 04:52:01.955523 18329 fs_manager.cc:359] Metadata directory not provided
I20260809 04:52:01.955624 18329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:52:01.955875 18329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 04:52:01 dist-test-slave-4tdj krb5kdc[17226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251121, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:52:01.969338 18329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "31465167c0e74663973f09d4d370bf48"
format_stamp: "Formatted at 2026-08-09 04:52:01 on dist-test-slave-4tdj"
server_key: "f1c5541b2a55396318897b4ef66b956c"
server_key_iv: "548577eb363d6d053180aff83d169712"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:01.970185 18329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "31465167c0e74663973f09d4d370bf48"
format_stamp: "Formatted at 2026-08-09 04:52:01 on dist-test-slave-4tdj"
server_key: "f1c5541b2a55396318897b4ef66b956c"
server_key_iv: "548577eb363d6d053180aff83d169712"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:01.975845 18329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 04:52:01.988762 18365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:01.990279 18329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 04:52:01.990514 18329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "31465167c0e74663973f09d4d370bf48"
format_stamp: "Formatted at 2026-08-09 04:52:01 on dist-test-slave-4tdj"
server_key: "f1c5541b2a55396318897b4ef66b956c"
server_key_iv: "548577eb363d6d053180aff83d169712"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:01.990722 18329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:52:02.013684 18329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:52:02.017767 18329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:52:02.018072 18329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:52:02.019209 18329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:52:02.020841 18329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:52:02.020918 18329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:02.021015 18329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:52:02.021065 18329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:02.057919 18329 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:36943
I20260809 04:52:02.057937 18539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:36943 every 8 connection(s)
I20260809 04:52:02.059670 18329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:52:02.063323 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 18329
I20260809 04:52:02.063522 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:52:02.064073 32116 external_mini_cluster.cc:1428] Setting key dbef7e31007f134932a35164dc41bf46
Aug 09 04:52:02 dist-test-slave-4tdj krb5kdc[17226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251121, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:52:02.085186 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.062478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:53553 (local address 127.31.93.62:32805)
0809 04:52:02.063001 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:02.063008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:02.064242 (+ 1234us) server_negotiation.cc:407] Connection header received
0809 04:52:02.066924 (+ 2682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:02.066929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:02.067001 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:02.067116 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:02.069130 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.070172 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.072554 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.073197 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.073342 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.077612 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:02.077620 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:02.077625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:02.077667 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:02.081403 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:02.082342 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:02.082348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:02.082352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:02.082536 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:02.083102 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:02.083107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:02.083111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:02.083505 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:02.083652 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:02.084510 (+ 858us) server_negotiation.cc:299] Negotiation successful
0809 04:52:02.084735 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":246,"threads_started":1}
I20260809 04:52:02.086390 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.063141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:32805 (local address 127.31.93.2:53553)
0809 04:52:02.064078 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:02.064127 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:02.066625 (+ 2498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:02.067373 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:02.067385 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:02.068066 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:02.068924 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:02.068950 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.070381 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.070387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:02.072365 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:02.072380 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.073457 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.073465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:02.073774 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.074609 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:02.074647 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:02.077354 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:02.081554 (+ 4200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:02.081563 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:02.081580 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:02.082165 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:02.082661 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:02.082665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:02.082669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:02.082951 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:02.083648 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:02.083657 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:02.084069 (+ 412us) client_negotiation.cc:769] Sending connection context
0809 04:52:02.084372 (+ 303us) client_negotiation.cc:240] Negotiation successful
0809 04:52:02.084894 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":391,"threads_started":1}
I20260809 04:52:02.088150 18540 heartbeater.cc:347] Connected to a master server at 127.31.93.62:32805
I20260809 04:52:02.088502 18540 heartbeater.cc:464] Registering TS with master...
I20260809 04:52:02.089488 18540 heartbeater.cc:511] Master 127.31.93.62:32805 requested a full tablet report, sending...
I20260809 04:52:02.091444 17997 ts_manager.cc:194] Registered new tserver with Master: 31465167c0e74663973f09d4d370bf48 (127.31.93.2:36943)
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
I20260809 04:52:02.092655 17997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:53553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:52:02.124856 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.93.20:56663/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:32805
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:42233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:52:02.418987 18318 heartbeater.cc:503] Master 127.31.93.62:32805 was elected leader, sending a full tablet report...
W20260809 04:52:02.477880 18547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:52:02.478156 18547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:52:02.478313 18547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:52:02.485551 18547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:52:02.485659 18547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:52:02.485816 18547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:52:02.495618 18547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:42233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.93.20:56663/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:32805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:52:02.496794 18547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:52:02.498065 18547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:52:02.511138 18561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:02.511250 18557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:02.511202 18554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:52:02.512233 18547 server_base.cc:1034] running on GCE node
I20260809 04:52:02.512764 18547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:52:02.513638 18547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:52:02.514830 18547 hybrid_clock.cc:648] HybridClock initialized: now 1786251122514774 us; error 46 us; skew 500 ppm
I20260809 04:52:02.515123 18547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:52:02 dist-test-slave-4tdj krb5kdc[17226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251122, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:52:02.518075 18547 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:52:02.519483 18547 webserver.cc:460] Webserver started at http://127.31.93.3:42761/ using document root <none> and password file <none>
I20260809 04:52:02.520087 18547 fs_manager.cc:359] Metadata directory not provided
I20260809 04:52:02.520246 18547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:52:02.520560 18547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 04:52:02 dist-test-slave-4tdj krb5kdc[17226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251122, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:52:02.535058 18547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "4622da01cc78406fb3b2fcb461f22a35"
format_stamp: "Formatted at 2026-08-09 04:52:02 on dist-test-slave-4tdj"
server_key: "006c8873efbd90177a72b419e2f7bc86"
server_key_iv: "8914954711892d41dd4097f11c23056f"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:02.535974 18547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "4622da01cc78406fb3b2fcb461f22a35"
format_stamp: "Formatted at 2026-08-09 04:52:02 on dist-test-slave-4tdj"
server_key: "006c8873efbd90177a72b419e2f7bc86"
server_key_iv: "8914954711892d41dd4097f11c23056f"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:02.543567 18547 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260809 04:52:02.560106 18572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:02.561594 18547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:52:02.561745 18547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "4622da01cc78406fb3b2fcb461f22a35"
format_stamp: "Formatted at 2026-08-09 04:52:02 on dist-test-slave-4tdj"
server_key: "006c8873efbd90177a72b419e2f7bc86"
server_key_iv: "8914954711892d41dd4097f11c23056f"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:02.561940 18547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:52:02.580744 18547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:52:02.584283 18547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:52:02.584578 18547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:52:02.585546 18547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:52:02.587067 18547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:52:02.587164 18547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:02.587302 18547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:52:02.587363 18547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:02.621680 18547 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:40055
I20260809 04:52:02.621739 18759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:40055 every 8 connection(s)
I20260809 04:52:02.622972 18547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:52:02.631246 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 18547
I20260809 04:52:02.631438 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:52:02.631990 32116 external_mini_cluster.cc:1428] Setting key 2a46a259c597ba3d50589e33c8dd96ac
Aug 09 04:52:02 dist-test-slave-4tdj krb5kdc[17226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251122, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:52:02.642285 18762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.625336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:34469 (local address 127.31.93.62:32805)
0809 04:52:02.625855 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:02.625861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:02.626822 (+ 961us) server_negotiation.cc:407] Connection header received
0809 04:52:02.628573 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:02.628577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:02.628658 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:02.628750 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:02.630071 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.630824 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.633048 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.633620 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.633718 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.637063 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:02.637067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:02.637070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:02.637108 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:02.639435 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:02.640104 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:02.640108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:02.640110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:02.640282 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:02.640712 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:02.640716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:02.640718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:02.640966 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:02.641105 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:02.641704 (+ 599us) server_negotiation.cc:299] Negotiation successful
0809 04:52:02.641922 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":217,"threads_started":1}
I20260809 04:52:02.643306 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.625908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:32805 (local address 127.31.93.3:34469)
0809 04:52:02.626668 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:02.626713 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:02.628338 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:02.628940 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:02.628947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:02.629358 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:02.629943 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:02.629960 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.630981 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.630987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:02.632900 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:02.632925 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.633822 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.633827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:02.634108 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.634743 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:02.634771 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:02.636870 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:02.639559 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:02.639568 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:02.639593 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:02.639987 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:02.640389 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:02.640394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:02.640398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:02.640603 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:02.641087 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:02.641095 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:02.641370 (+ 275us) client_negotiation.cc:769] Sending connection context
0809 04:52:02.641599 (+ 229us) client_negotiation.cc:240] Negotiation successful
0809 04:52:02.642071 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":352,"threads_started":1}
I20260809 04:52:02.644520 18760 heartbeater.cc:347] Connected to a master server at 127.31.93.62:32805
I20260809 04:52:02.644796 18760 heartbeater.cc:464] Registering TS with master...
I20260809 04:52:02.645568 18760 heartbeater.cc:511] Master 127.31.93.62:32805 requested a full tablet report, sending...
I20260809 04:52:02.647213 17995 ts_manager.cc:194] Registered new tserver with Master: 4622da01cc78406fb3b2fcb461f22a35 (127.31.93.3:40055)
I20260809 04:52:02.648136 17995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:34469
I20260809 04:52:02.659139 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:52:02.676226 18762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.664361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.31.93.62:32805)
0809 04:52:02.664511 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:02.664516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:02.665014 (+ 498us) server_negotiation.cc:407] Connection header received
0809 04:52:02.665161 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:02.665165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:02.665236 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:02.665308 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:02.666851 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.667551 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.668761 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.669211 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.669298 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.671544 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:02.671549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:02.671552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:02.671581 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:02.674062 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:02.674661 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:02.674664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:02.674666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:02.674794 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:02.675061 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:02.675064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:02.675066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:02.675358 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:02.675547 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:02.675769 (+ 222us) server_negotiation.cc:299] Negotiation successful
0809 04:52:02.675943 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 04:52:02.680269 17995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:52:02.688113 17995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:52:02.724758 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.714915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.31.93.1:36161)
0809 04:52:02.716937 (+ 2022us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:02.716945 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:02.716971 (+ 26us) server_negotiation.cc:407] Connection header received
0809 04:52:02.717049 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:02.717054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:02.717350 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:02.717593 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:52:02.718433 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.720377 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.722369 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.723694 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.723817 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.724062 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:02.724347 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 04:52:02.724474 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1779,"thread_start_us":1509,"threads_started":1}
I20260809 04:52:02.724920 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.700618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:36943 (local address 127.0.0.1:49456)
0809 04:52:02.713020 (+ 12402us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:02.713064 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:02.713270 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:02.716800 (+ 3530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:02.716805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:02.716837 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:52:02.718040 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:02.718055 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.720196 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.720201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:02.722422 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:02.722436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.723996 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.724002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:02.724187 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.724212 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 04:52:02.724284 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 04:52:02.724493 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12141,"mutex_wait_us":57,"thread_start_us":13063,"threads_started":1}
I20260809 04:52:02.725309 18782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.700868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:40055 (local address 127.0.0.1:51866)
0809 04:52:02.714147 (+ 13279us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:02.714164 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:02.714271 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:02.716371 (+ 2100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:02.716379 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:02.716409 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:52:02.717417 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:02.717429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.719398 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.719402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:02.721983 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:02.721998 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.724484 (+ 2486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.724489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:02.724663 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.724694 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 04:52:02.724785 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 04:52:02.724917 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13145,"mutex_wait_us":43,"thread_start_us":11724,"threads_started":1}
I20260809 04:52:02.725546 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.701570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.31.93.3:40055)
0809 04:52:02.715259 (+ 13689us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:02.715267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:02.715311 (+ 44us) server_negotiation.cc:407] Connection header received
0809 04:52:02.715607 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:02.715615 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:02.715844 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:02.716046 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:52:02.717621 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.719244 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.722162 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.723685 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.723826 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.724062 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:02.724868 (+ 806us) server_negotiation.cc:299] Negotiation successful
0809 04:52:02.725054 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13490,"thread_start_us":14555,"threads_started":1}
I20260809 04:52:02.725764 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.714401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:36161 (local address 127.0.0.1:54014)
0809 04:52:02.715876 (+ 1475us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:02.715896 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:02.716029 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:02.717586 (+ 1557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:02.717590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:02.717616 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:52:02.718238 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:02.718252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.720540 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.720546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:02.722171 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:02.722186 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.723895 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.723902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:02.724110 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.724180 (+ 70us) client_negotiation.cc:769] Sending connection context
0809 04:52:02.724311 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 04:52:02.725294 (+ 983us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1348,"mutex_wait_us":885,"thread_start_us":1190,"threads_started":1}
I20260809 04:52:02.726449 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.700879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.31.93.2:36943)
0809 04:52:02.714779 (+ 13900us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:02.714789 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:02.714824 (+ 35us) server_negotiation.cc:407] Connection header received
0809 04:52:02.714933 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:02.714942 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:02.715229 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:02.716806 (+ 1577us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:52:02.718319 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.720065 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.722602 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.723779 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.725651 (+ 1872us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.725828 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:02.725897 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 04:52:02.726076 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13698,"thread_start_us":13567,"threads_started":1}
I20260809 04:52:02.729789 18225 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8bd52be9a5495e82b3fbefdf367ee1 (DEFAULT_TABLE table=test-table [id=533d92c5f1394d61b770153b36ede9fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:52:02.730095 18438 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8bd52be9a5495e82b3fbefdf367ee1 (DEFAULT_TABLE table=test-table [id=533d92c5f1394d61b770153b36ede9fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:52:02.729938 18651 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8bd52be9a5495e82b3fbefdf367ee1 (DEFAULT_TABLE table=test-table [id=533d92c5f1394d61b770153b36ede9fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:52:02.731134 18225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8bd52be9a5495e82b3fbefdf367ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:52:02.731451 18438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8bd52be9a5495e82b3fbefdf367ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:52:02.731454 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8bd52be9a5495e82b3fbefdf367ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:52:02.742962 18794 tablet_bootstrap.cc:492] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec: Bootstrap starting.
I20260809 04:52:02.747112 18796 tablet_bootstrap.cc:492] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48: Bootstrap starting.
I20260809 04:52:02.747720 18794 tablet_bootstrap.cc:654] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec: Neither blocks nor log segments found. Creating new log.
I20260809 04:52:02.749303 18794 log.cc:824] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec: Log is configured to *not* fsync() on all Append() calls
I20260809 04:52:02.750780 18795 tablet_bootstrap.cc:492] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35: Bootstrap starting.
I20260809 04:52:02.750934 18796 tablet_bootstrap.cc:654] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48: Neither blocks nor log segments found. Creating new log.
I20260809 04:52:02.752228 18796 log.cc:824] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48: Log is configured to *not* fsync() on all Append() calls
I20260809 04:52:02.752583 18794 tablet_bootstrap.cc:492] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec: No bootstrap required, opened a new log
I20260809 04:52:02.752918 18794 ts_tablet_manager.cc:1397] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260809 04:52:02.754905 18795 tablet_bootstrap.cc:654] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35: Neither blocks nor log segments found. Creating new log.
I20260809 04:52:02.755641 18796 tablet_bootstrap.cc:492] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48: No bootstrap required, opened a new log
I20260809 04:52:02.755919 18796 ts_tablet_manager.cc:1397] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 04:52:02.756203 18795 log.cc:824] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35: Log is configured to *not* fsync() on all Append() calls
I20260809 04:52:02.760075 18795 tablet_bootstrap.cc:492] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35: No bootstrap required, opened a new log
I20260809 04:52:02.760473 18795 ts_tablet_manager.cc:1397] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260809 04:52:02.762531 18794 raft_consensus.cc:348] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31465167c0e74663973f09d4d370bf48" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36943 } } peers { permanent_uuid: "4622da01cc78406fb3b2fcb461f22a35" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40055 } } peers { permanent_uuid: "4529e4ac5d7e422c9d030390fd1922ec" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 36161 } }
I20260809 04:52:02.762944 18794 raft_consensus.cc:374] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:52:02.763064 18794 raft_consensus.cc:729] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4529e4ac5d7e422c9d030390fd1922ec, State: Initialized, Role: FOLLOWER
I20260809 04:52:02.763641 18794 consensus_queue.cc:260] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31465167c0e74663973f09d4d370bf48" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36943 } } peers { permanent_uuid: "4622da01cc78406fb3b2fcb461f22a35" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40055 } } peers { permanent_uuid: "4529e4ac5d7e422c9d030390fd1922ec" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 36161 } }
I20260809 04:52:02.766582 18794 ts_tablet_manager.cc:1428] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260809 04:52:02.767050 18796 raft_consensus.cc:348] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31465167c0e74663973f09d4d370bf48" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36943 } } peers { permanent_uuid: "4622da01cc78406fb3b2fcb461f22a35" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40055 } } peers { permanent_uuid: "4529e4ac5d7e422c9d030390fd1922ec" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 36161 } }
I20260809 04:52:02.767489 18796 raft_consensus.cc:374] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:52:02.767611 18796 raft_consensus.cc:729] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31465167c0e74663973f09d4d370bf48, State: Initialized, Role: FOLLOWER
I20260809 04:52:02.768214 18796 consensus_queue.cc:260] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31465167c0e74663973f09d4d370bf48" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36943 } } peers { permanent_uuid: "4622da01cc78406fb3b2fcb461f22a35" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40055 } } peers { permanent_uuid: "4529e4ac5d7e422c9d030390fd1922ec" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 36161 } }
I20260809 04:52:02.769550 18540 heartbeater.cc:503] Master 127.31.93.62:32805 was elected leader, sending a full tablet report...
I20260809 04:52:02.769995 18796 ts_tablet_manager.cc:1428] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260809 04:52:02.771477 18795 raft_consensus.cc:348] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31465167c0e74663973f09d4d370bf48" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36943 } } peers { permanent_uuid: "4622da01cc78406fb3b2fcb461f22a35" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40055 } } peers { permanent_uuid: "4529e4ac5d7e422c9d030390fd1922ec" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 36161 } }
I20260809 04:52:02.771847 18795 raft_consensus.cc:374] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:52:02.771961 18795 raft_consensus.cc:729] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4622da01cc78406fb3b2fcb461f22a35, State: Initialized, Role: FOLLOWER
I20260809 04:52:02.772627 18795 consensus_queue.cc:260] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31465167c0e74663973f09d4d370bf48" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36943 } } peers { permanent_uuid: "4622da01cc78406fb3b2fcb461f22a35" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40055 } } peers { permanent_uuid: "4529e4ac5d7e422c9d030390fd1922ec" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 36161 } }
I20260809 04:52:02.773758 18760 heartbeater.cc:503] Master 127.31.93.62:32805 was elected leader, sending a full tablet report...
I20260809 04:52:02.774260 18795 ts_tablet_manager.cc:1428] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260809 04:52:02.810902 18806 raft_consensus.cc:482] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:52:02.811182 18806 raft_consensus.cc:504] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31465167c0e74663973f09d4d370bf48" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36943 } } peers { permanent_uuid: "4622da01cc78406fb3b2fcb461f22a35" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40055 } } peers { permanent_uuid: "4529e4ac5d7e422c9d030390fd1922ec" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 36161 } }
W20260809 04:52:02.811837 18541 tablet.cc:2369] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:52:02.812911 18806 leader_election.cc:288] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31465167c0e74663973f09d4d370bf48 (127.31.93.2:36943), 4529e4ac5d7e422c9d030390fd1922ec (127.31.93.1:36161)
I20260809 04:52:02.819129 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.813133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:36943 (local address 127.31.93.3:32883)
0809 04:52:02.813297 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:02.813314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:02.813432 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:02.813914 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:02.813919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:02.813958 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:52:02.814176 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:02.814189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.815444 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.815448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:02.817233 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:02.817248 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.818603 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.818607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:02.818740 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.818766 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 04:52:02.818837 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 04:52:02.818926 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":1}
I20260809 04:52:02.819272 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.813308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:32883 (local address 127.31.93.2:36943)
0809 04:52:02.813522 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:02.813528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:02.813545 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:52:02.813602 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:02.813605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:02.813663 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:02.813740 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:52:02.814362 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.815306 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.817384 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.818465 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.818593 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.818687 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:02.818887 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 04:52:02.819016 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 04:52:02.820124 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.813774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:33155 (local address 127.31.93.1:36161)
0809 04:52:02.813985 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:02.813991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:02.814382 (+ 391us) server_negotiation.cc:407] Connection header received
0809 04:52:02.814515 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:02.814520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:02.814607 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:02.814702 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:52:02.815168 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.815957 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.817988 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.819154 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.819277 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.819398 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:02.819831 (+ 433us) server_negotiation.cc:299] Negotiation successful
0809 04:52:02.819954 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 04:52:02.820272 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8bd52be9a5495e82b3fbefdf367ee1" candidate_uuid: "4622da01cc78406fb3b2fcb461f22a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31465167c0e74663973f09d4d370bf48" is_pre_election: true
I20260809 04:52:02.820818 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.813470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:36161 (local address 127.31.93.3:33155)
0809 04:52:02.814244 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:02.814261 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:02.814404 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:02.814703 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:02.814706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:02.814721 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:52:02.815033 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:02.815041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.816127 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.816133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:02.817847 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:02.817858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.819528 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.819534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:02.819650 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.819675 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 04:52:02.820307 (+ 632us) client_negotiation.cc:240] Negotiation successful
0809 04:52:02.820426 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":433,"threads_started":1}
I20260809 04:52:02.820819 18472 raft_consensus.cc:2454] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4622da01cc78406fb3b2fcb461f22a35 in term 0.
I20260809 04:52:02.821502 18576 leader_election.cc:302] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31465167c0e74663973f09d4d370bf48, 4622da01cc78406fb3b2fcb461f22a35; no voters:
I20260809 04:52:02.821866 18806 raft_consensus.cc:2781] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:52:02.821983 18806 raft_consensus.cc:482] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:52:02.821846 18249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8bd52be9a5495e82b3fbefdf367ee1" candidate_uuid: "4622da01cc78406fb3b2fcb461f22a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4529e4ac5d7e422c9d030390fd1922ec" is_pre_election: true
I20260809 04:52:02.822126 18806 raft_consensus.cc:3037] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:52:02.822402 18249 raft_consensus.cc:2454] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4622da01cc78406fb3b2fcb461f22a35 in term 0.
I20260809 04:52:02.824536 18806 raft_consensus.cc:504] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31465167c0e74663973f09d4d370bf48" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36943 } } peers { permanent_uuid: "4622da01cc78406fb3b2fcb461f22a35" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40055 } } peers { permanent_uuid: "4529e4ac5d7e422c9d030390fd1922ec" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 36161 } }
I20260809 04:52:02.825090 18806 leader_election.cc:288] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [CANDIDATE]: Term 1 election: Requested vote from peers 31465167c0e74663973f09d4d370bf48 (127.31.93.2:36943), 4529e4ac5d7e422c9d030390fd1922ec (127.31.93.1:36161)
I20260809 04:52:02.825423 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8bd52be9a5495e82b3fbefdf367ee1" candidate_uuid: "4622da01cc78406fb3b2fcb461f22a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31465167c0e74663973f09d4d370bf48"
I20260809 04:52:02.825628 18472 raft_consensus.cc:3037] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:52:02.825601 18249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8bd52be9a5495e82b3fbefdf367ee1" candidate_uuid: "4622da01cc78406fb3b2fcb461f22a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4529e4ac5d7e422c9d030390fd1922ec"
I20260809 04:52:02.825802 18249 raft_consensus.cc:3037] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:52:02.828939 18249 raft_consensus.cc:2454] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4622da01cc78406fb3b2fcb461f22a35 in term 1.
I20260809 04:52:02.828939 18472 raft_consensus.cc:2454] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4622da01cc78406fb3b2fcb461f22a35 in term 1.
I20260809 04:52:02.829316 18576 leader_election.cc:302] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4529e4ac5d7e422c9d030390fd1922ec, 4622da01cc78406fb3b2fcb461f22a35; no voters:
I20260809 04:52:02.829569 18806 raft_consensus.cc:2781] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:52:02.829999 18806 raft_consensus.cc:686] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [term 1 LEADER]: Becoming Leader. State: Replica: 4622da01cc78406fb3b2fcb461f22a35, State: Running, Role: LEADER
I20260809 04:52:02.830518 18806 consensus_queue.cc:237] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31465167c0e74663973f09d4d370bf48" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36943 } } peers { permanent_uuid: "4622da01cc78406fb3b2fcb461f22a35" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40055 } } peers { permanent_uuid: "4529e4ac5d7e422c9d030390fd1922ec" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 36161 } }
I20260809 04:52:02.834898 17995 catalog_manager.cc:5495] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4622da01cc78406fb3b2fcb461f22a35 (127.31.93.3). New cstate: current_term: 1 leader_uuid: "4622da01cc78406fb3b2fcb461f22a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31465167c0e74663973f09d4d370bf48" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4622da01cc78406fb3b2fcb461f22a35" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4529e4ac5d7e422c9d030390fd1922ec" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 36161 } health_report { overall_health: UNKNOWN } } }
I20260809 04:52:02.856535 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.851553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.31.93.3:40055)
0809 04:52:02.851718 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:02.851723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:02.851781 (+ 58us) server_negotiation.cc:407] Connection header received
0809 04:52:02.851972 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:02.851976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:02.852052 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:02.852167 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:52:02.852497 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.853188 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.854728 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.855124 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.855223 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.855688 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:52:02.856139 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:52:02.856239 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:02.856300 (+ 61us) server_negotiation.cc:299] Negotiation successful
0809 04:52:02.856358 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 04:52:02.864722 18472 raft_consensus.cc:1261] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48 [term 1 FOLLOWER]: Refusing update from remote peer 4622da01cc78406fb3b2fcb461f22a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:52:02.864729 18249 raft_consensus.cc:1261] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec [term 1 FOLLOWER]: Refusing update from remote peer 4622da01cc78406fb3b2fcb461f22a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:52:02.865646 18806 consensus_queue.cc:1035] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4529e4ac5d7e422c9d030390fd1922ec" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 36161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:52:02.865958 18813 consensus_queue.cc:1035] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31465167c0e74663973f09d4d370bf48" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 04:52:02.872768 18319 tablet.cc:2369] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:52:02.874576 18761 tablet.cc:2369] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:52:02.877148 18817 mvcc.cc:205] Tried to move back new op lower bound from 7316484599244148736 to 7316484599117660160. Current Snapshot: MvccSnapshot[applied={T|T < 7316484599244148736}]
I20260809 04:52:02.894009 17995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:52:02.894284 17995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:52:02.897864 17995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 137fcc7ead854c6088a4b6c3c61232cb: Sending DeleteTablet for 3 replicas of tablet 5b8bd52be9a5495e82b3fbefdf367ee1
I20260809 04:52:02.898958 18438 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8bd52be9a5495e82b3fbefdf367ee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:52:02 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:49456
I20260809 04:52:02.899163 18225 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8bd52be9a5495e82b3fbefdf367ee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:52:02 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:54014
I20260809 04:52:02.899487 18651 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8bd52be9a5495e82b3fbefdf367ee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:52:02 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:51866
I20260809 04:52:02.902212 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 18116
I20260809 04:52:02.913800 18830 tablet_replica.cc:330] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec: stopping tablet replica
I20260809 04:52:02.917361 18830 raft_consensus.cc:2229] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:52:02.918087 18830 raft_consensus.cc:2258] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:52:02.918941 18832 tablet_replica.cc:330] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35: stopping tablet replica
I20260809 04:52:02.919660 18828 tablet_replica.cc:330] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48: stopping tablet replica
I20260809 04:52:02.920089 18828 raft_consensus.cc:2229] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:52:02.920640 18828 raft_consensus.cc:2258] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:52:02.921376 18832 raft_consensus.cc:2229] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:52:02.922948 18828 ts_tablet_manager.cc:1905] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:52:02.922955 18830 ts_tablet_manager.cc:1905] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:52:02.923671 18832 raft_consensus.cc:2258] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 09 04:52:02 dist-test-slave-4tdj krb5kdc[17226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:52:02.925932 18832 ts_tablet_manager.cc:1905] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:52:02.929975 18828 ts_tablet_manager.cc:1918] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:52:02.930208 18828 log.cc:1196] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/5b8bd52be9a5495e82b3fbefdf367ee1
I20260809 04:52:02.930814 18828 ts_tablet_manager.cc:1939] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 31465167c0e74663973f09d4d370bf48: Deleting consensus metadata
I20260809 04:52:02.932716 17967 catalog_manager.cc:4841] TS 31465167c0e74663973f09d4d370bf48 (127.31.93.2:36943): tablet 5b8bd52be9a5495e82b3fbefdf367ee1 (table test-table [id=533d92c5f1394d61b770153b36ede9fb]) successfully deleted
I20260809 04:52:02.932927 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:02.903505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.31.93.1:36161)
0809 04:52:02.903684 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:02.903689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:02.908466 (+ 4777us) server_negotiation.cc:407] Connection header received
0809 04:52:02.909295 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:02.909301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:02.909410 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:02.909483 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:02.914104 (+ 4621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.915351 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.919175 (+ 3824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:02.919711 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:02.919849 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:02.925498 (+ 5649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:02.925512 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:02.925529 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:02.925585 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:02.929364 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:02.930248 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:02.930257 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:02.930260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:02.930452 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:02.930914 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:02.930919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:02.930922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:02.931399 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:02.931694 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:02.932312 (+ 618us) server_negotiation.cc:299] Negotiation successful
0809 04:52:02.932525 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 04:52:02.933988 18830 ts_tablet_manager.cc:1918] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:52:02.934219 18830 log.cc:1196] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/5b8bd52be9a5495e82b3fbefdf367ee1
I20260809 04:52:02.934244 18311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:52:02.941852 18832 ts_tablet_manager.cc:1918] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:52:02.942024 18832 log.cc:1196] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/5b8bd52be9a5495e82b3fbefdf367ee1
I20260809 04:52:02.942617 18832 ts_tablet_manager.cc:1939] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4622da01cc78406fb3b2fcb461f22a35: Deleting consensus metadata
I20260809 04:52:02.944518 17970 catalog_manager.cc:4841] TS 4622da01cc78406fb3b2fcb461f22a35 (127.31.93.3:40055): tablet 5b8bd52be9a5495e82b3fbefdf367ee1 (table test-table [id=533d92c5f1394d61b770153b36ede9fb]) successfully deleted
I20260809 04:52:03.029516 18830 ts_tablet_manager.cc:1939] T 5b8bd52be9a5495e82b3fbefdf367ee1 P 4529e4ac5d7e422c9d030390fd1922ec: Deleting consensus metadata
I20260809 04:52:03.031893 17967 catalog_manager.cc:4841] TS 4529e4ac5d7e422c9d030390fd1922ec (127.31.93.1:36161): tablet 5b8bd52be9a5495e82b3fbefdf367ee1 (table test-table [id=533d92c5f1394d61b770153b36ede9fb]) successfully deleted
I20260809 04:52:03.032356 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 18116
I20260809 04:52:03.060027 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 18329
Aug 09 04:52:03 dist-test-slave-4tdj krb5kdc[17226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:52:03.075636 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:03.060811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49458 (local address 127.31.93.2:36943)
0809 04:52:03.060996 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:03.061001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:03.061110 (+ 109us) server_negotiation.cc:407] Connection header received
0809 04:52:03.061290 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:03.061295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:03.061380 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:03.061466 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:03.062837 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:03.063588 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:03.064910 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:03.065324 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:03.065435 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:03.069814 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:03.069825 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:03.069839 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:03.069871 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:03.072791 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:03.073484 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:03.073491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:03.073494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:03.073637 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:03.074013 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:03.074017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:03.074019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:03.074445 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:03.074732 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:03.075055 (+ 323us) server_negotiation.cc:299] Negotiation successful
0809 04:52:03.075279 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 04:52:03.076318 18533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:52:03.187422 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 18329
I20260809 04:52:03.210230 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 18547
Aug 09 04:52:03 dist-test-slave-4tdj krb5kdc[17226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:52:03.228084 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:03.211132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.31.93.3:40055)
0809 04:52:03.211286 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:03.211291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:03.211311 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:52:03.211404 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:03.211408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:03.211472 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:03.211556 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:03.213593 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:03.214569 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:03.216474 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:03.216941 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:03.217052 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:03.222131 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:03.222144 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:03.222160 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:03.222215 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:03.225162 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:03.225796 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:03.225805 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:03.225808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:03.225998 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:03.226405 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:03.226411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:03.226414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:03.226799 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:03.227111 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:03.227411 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 04:52:03.227630 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:52:03.228883 18751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:52:03.349838 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 18547
I20260809 04:52:03.373688 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 17933
I20260809 04:52:03.374852 18041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:52:03.485232 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 17933
I20260809 04:52:03.512844 32116 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 04:52:03.856796 32116 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 04:52:03.857085 32116 mini_ranger.cc:66] Stopping Ranger...
I20260809 04:52:04.201840 32116 mini_ranger.cc:68] Stopped Ranger
2026-08-09 04:52:04.202 UTC [17261] LOG: received smart shutdown request
2026-08-09 04:52:04.205 UTC [17261] LOG: background worker "logical replication launcher" (PID 17275) exited with exit code 1
2026-08-09 04:52:04.205 UTC [17268] LOG: shutting down
2026-08-09 04:52:04.329 UTC [17261] LOG: database system is shut down
2026-08-09T04:52:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48652 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:52:04 dist-test-slave-4tdj krb5kdc[18859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:52:04 dist-test-slave-4tdj krb5kdc[18859](info): set up 2 sockets
Aug 09 04:52:04 dist-test-slave-4tdj krb5kdc[18859](info): commencing operation
krb5kdc: starting...
W20260809 04:52:05.632844 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:52:05 dist-test-slave-4tdj krb5kdc[18859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:52:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:52:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.93.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.93.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.93.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/rangeradmin_127.31.93.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.93.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/rangeradmin_127.31.93.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.93.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.93.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.93.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/rangerlookup_127.31.93.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.93.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/rangerlookup_127.31.93.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.93.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.93.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.93.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/HTTP_127.31.93.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.93.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/HTTP_127.31.93.20@KRBTEST.COM.keytab.
I20260809 04:52:05.956434 32116 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAAy6Yk/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/postgres -l logfile start
2026-08-09 04:52:08.069 UTC [18909] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 04:52:08.069 UTC [18909] LOG: listening on IPv4 address "127.31.93.20", port 52343
2026-08-09 04:52:08.071 UTC [18909] LOG: listening on Unix socket "/tmp/.s.PGSQL.52343"
2026-08-09 04:52:08.091 UTC [18915] LOG: database system was shut down at 2026-08-09 04:52:06 UTC
2026-08-09 04:52:08.094 UTC [18909] LOG: database system is ready to accept connections
I20260809 04:52:09.204126 32116 mini_postgres.cc:96] Postgres bound to 52343
2026-08-09 04:52:09.229 UTC [18929] FATAL: database "slave" does not exist
127.31.93.20:52343 - accepting connections
I20260809 04:52:09.230554 32116 mini_ranger.cc:161] Starting Ranger...
I20260809 04:52:09.257386 32116 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 04:52:09.617671 32116 mini_ranger.cc:87] Created ranger Postgres database
I20260809 04:52:09.617798 32116 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-admin
2026-08-09 04:52:09,900 [I] DB FLAVOR :POSTGRES
2026-08-09 04:52:09,901 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 04:52:09,901 [I] Checking connection..
2026-08-09 04:52:09,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 04:52:10,157 [I] Checking connection passed.
2026-08-09 04:52:10,158 [I] --------- Verifying version history table ---------
2026-08-09 04:52:10,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 04:52:10,412 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 04:52:10,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 04:52:10,666 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 04:52:10,666 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 04:52:10,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 04:52:10.898 UTC [19036] WARNING: there is no transaction in progress
2026-08-09 04:52:10,906 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 04:52:10,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 04:52:11,165 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 04:52:11,166 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 04:52:11,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 04:52:11,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 04:52:11,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4tdj.c.gcp-upstream.internal','N') ;"
2026-08-09 04:52:11,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 04:52:12,144 [I] Table x_portal_user does not exist in database ranger
2026-08-09 04:52:12,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 04:52:12,385 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 04:52:12,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 04:52:12,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 04:52:12,879 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 04:52:12,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 04:52:13.722 UTC [19202] WARNING: there is no transaction in progress
2026-08-09 04:52:13.737 UTC [19202] WARNING: there is no transaction in progress
2026-08-09 04:52:13.751 UTC [19202] WARNING: there is no transaction in progress
2026-08-09 04:52:13.764 UTC [19202] WARNING: there is no transaction in progress
2026-08-09 04:52:13.778 UTC [19202] WARNING: there is no transaction in progress
2026-08-09 04:52:13.792 UTC [19202] WARNING: there is no transaction in progress
2026-08-09 04:52:13.840 UTC [19202] WARNING: there is no transaction in progress
2026-08-09 04:52:13.846 UTC [19202] WARNING: there is no transaction in progress
2026-08-09 04:52:13.858 UTC [19202] WARNING: there is no transaction in progress
2026-08-09 04:52:13.867 UTC [19202] WARNING: there is no transaction in progress
2026-08-09 04:52:13.876 UTC [19202] WARNING: there is no transaction in progress
2026-08-09 04:52:13.887 UTC [19202] WARNING: there is no transaction in progress
2026-08-09 04:52:13.896 UTC [19202] WARNING: there is no transaction in progress
2026-08-09 04:52:13.901 UTC [19202] WARNING: there is no transaction in progress
2026-08-09 04:52:13.906 UTC [19202] WARNING: there is no transaction in progress
2026-08-09 04:52:13.912 UTC [19202] WARNING: there is no transaction in progress
2026-08-09 04:52:14,433 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 04:52:14,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 04:52:14,680 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 04:52:14,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 04:52:14,933 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 04:52:14,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 04:52:15,182 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 04:52:15,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 04:52:15,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 04:52:15,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4tdj.c.gcp-upstream.internal';"
2026-08-09 04:52:15,896 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 04:52:15,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 04:52:16,141 [I] DB_PATCHES have already been applied
I20260809 04:52:16.147663 32116 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAAy6Yk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 04:52:16.147747 32116 mini_ranger.cc:193] Using host: 127.31.93.20
I20260809 04:52:16.165496 32116 mini_ranger.cc:227] Ranger admin URL: http://127.31.93.20:36369
Aug 09, 2026 4:52:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 4:52:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 4:52:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 4:52:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 4:52:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 4:52:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 4:52:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 04:52:17.390213 32116 mini_ranger.cc:160] Time spent starting Ranger: real 8.160s user 0.000s sys 0.048s
Aug 09, 2026 4:52:37 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 4:52:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 09, 2026 4:52:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 4:52:37 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 4:52:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 4:52:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 4:52:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 4:52:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 4:52:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 4:52:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 4:52:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 4:52:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 4:52:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 4:52:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 4:52:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 4:52:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 4:52:37 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 4:52:38 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 04:52:38.646180 32116 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.93.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.93.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.93.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/rangerkms_127.31.93.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.93.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/rangerkms_127.31.93.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.93.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.93.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.93.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/HTTP_127.31.93.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.93.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/HTTP_127.31.93.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 04:52:38 dist-test-slave-4tdj krb5kdc[18859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251158, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 04:52:38.814735 32116 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 04:52:38.833128 32116 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 04:52:39.488662 32116 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 04:52:39.488773 32116 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-kms
I20260809 04:52:39.488860 32116 mini_ranger_kms.cc:227] Using postgres at 127.31.93.20:52343
2026-08-09 04:52:39,833 [I] DB FLAVOR :POSTGRES
2026-08-09 04:52:39,833 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 04:52:39,833 [I] Checking connection
2026-08-09 04:52:39,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 04:52:40,059 [I] connection success
2026-08-09 04:52:40,059 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 04:52:40,059 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 04:52:40,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 04:52:40,289 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 04:52:40,289 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 04:52:40,289 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 04:52:40,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.93.20:52343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 04:52:40,557 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 04:52:41.000535 32116 mini_ranger_kms.cc:310] Created kms service
I20260809 04:52:41.198854 32116 mini_ranger_kms.cc:326] Created kudu user
I20260809 04:52:41.243714 32116 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 04:52:41 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251158, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:52:41.625998 32116 mini_ranger_kms.cc:384] Added ranger policy
I20260809 04:52:41.626392 32116 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAAy6Yk/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAAy6Yk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 04:52:41.626467 32116 mini_ranger_kms.cc:240] Using host: 127.31.93.20
I20260809 04:52:41.644585 32116 mini_ranger_kms.cc:276] Ranger KMS PID: 19466
I20260809 04:52:41.644704 32116 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.93.20:51503
Aug 09, 2026 4:52:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 4:52:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 4:52:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 4:52:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 4:52:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 4:52:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 04:52:43 dist-test-slave-4tdj krb5kdc[18859](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786251163, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.93.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:52:44.229390 32116 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.415s user 0.079s sys 0.289s
I20260809 04:52:44.229657 32116 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 04:52:44.229841 32116 mini_ranger_kms.cc:401] 127.31.93.20:51503/kms/v1/keys
Aug 09 04:52:44 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251163, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.93.20@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
Aug 09 04:52:44 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251163, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.93.20@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
Aug 09 04:52:46 dist-test-slave-4tdj krb5kdc[18859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:52:46.959649 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:41443
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:41443
--ranger_config_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.93.20:51503/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:52:47.293541 19541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:52:47.293805 19541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:52:47.293938 19541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:52:47.300870 19541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:52:47.300971 19541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:52:47.301052 19541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:52:47.301131 19541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:52:47.301194 19541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:52:47.310655 19541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.93.20:51503/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:41443
--ranger_config_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:41443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:52:47.311762 19541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:52:47.313102 19541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:52:47.324681 19547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:47.324803 19548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:47.324715 19550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:52:47.325341 19541 server_base.cc:1034] running on GCE node
I20260809 04:52:47.326076 19541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:52:47.327386 19541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:52:47.328603 19541 hybrid_clock.cc:648] HybridClock initialized: now 1786251167328554 us; error 35 us; skew 500 ppm
I20260809 04:52:47.328933 19541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:52:47 dist-test-slave-4tdj krb5kdc[18859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251167, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:52:47.331873 19541 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:52:47.333177 19541 webserver.cc:460] Webserver started at http://127.31.93.62:34563/ using document root <none> and password file <none>
I20260809 04:52:47.333792 19541 fs_manager.cc:359] Metadata directory not provided
I20260809 04:52:47.333910 19541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:52:47.334193 19541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 04:52:47 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251167, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:52:47.392732 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "9f5b0087f2444520aa6722bbe2335bca"
format_stamp: "Formatted at 2026-08-09 04:52:47 on dist-test-slave-4tdj"
server_key: "78f06485fa3c8e75692a6fe5b9b10c1c"
server_key_iv: "b9b4013ed160722b4cb1f13c00acae96"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:47.393584 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "9f5b0087f2444520aa6722bbe2335bca"
format_stamp: "Formatted at 2026-08-09 04:52:47 on dist-test-slave-4tdj"
server_key: "78f06485fa3c8e75692a6fe5b9b10c1c"
server_key_iv: "b9b4013ed160722b4cb1f13c00acae96"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:47.399144 19541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 04:52:47.418397 19558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:47.420115 19541 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:52:47.420316 19541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "9f5b0087f2444520aa6722bbe2335bca"
format_stamp: "Formatted at 2026-08-09 04:52:47 on dist-test-slave-4tdj"
server_key: "78f06485fa3c8e75692a6fe5b9b10c1c"
server_key_iv: "b9b4013ed160722b4cb1f13c00acae96"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:47.420585 19541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:52:47.450500 19541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:52:47.454164 19541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:52:47.454501 19541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:52:47.472918 19541 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:41443
I20260809 04:52:47.472955 19639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:41443 every 8 connection(s)
I20260809 04:52:47.474337 19541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:52:47.477975 19640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:52:47.483117 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 19541
I20260809 04:52:47.483335 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:52:47.484249 32116 external_mini_cluster.cc:1428] Setting key 52da4eafd016a45f430045cf939b2636
I20260809 04:52:47.489282 19640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca: Bootstrap starting.
I20260809 04:52:47.492215 19640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca: Neither blocks nor log segments found. Creating new log.
I20260809 04:52:47.493183 19640 log.cc:824] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca: Log is configured to *not* fsync() on all Append() calls
I20260809 04:52:47.495925 19640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca: No bootstrap required, opened a new log
Aug 09 04:52:47 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:52:47.504882 19640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5b0087f2444520aa6722bbe2335bca" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41443 } }
I20260809 04:52:47.505330 19640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:52:47.505492 19640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f5b0087f2444520aa6722bbe2335bca, State: Initialized, Role: FOLLOWER
I20260809 04:52:47.506181 19640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5b0087f2444520aa6722bbe2335bca" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41443 } }
I20260809 04:52:47.506506 19640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:52:47.506661 19640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:52:47.506836 19640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:52:47.506951 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:47.486400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.31.93.62:41443)
0809 04:52:47.487231 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:47.487242 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:47.487292 (+ 50us) server_negotiation.cc:407] Connection header received
0809 04:52:47.489083 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:47.489107 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:47.489501 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:47.489854 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:47.491345 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:47.492706 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:47.494053 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:47.494782 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:47.494956 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:47.498987 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:47.498997 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:47.499015 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:47.499059 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:47.502740 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:47.503396 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:47.503404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:47.503413 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:47.503604 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:47.503953 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:47.503958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:47.503961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:47.504653 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:47.504943 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:47.505242 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 04:52:47.505626 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":371,"threads_started":1}
I20260809 04:52:47.509057 19640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5b0087f2444520aa6722bbe2335bca" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41443 } }
I20260809 04:52:47.509563 19640 leader_election.cc:302] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f5b0087f2444520aa6722bbe2335bca; no voters:
I20260809 04:52:47.510157 19640 leader_election.cc:288] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:52:47.510274 19646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:52:47.510919 19646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 1 LEADER]: Becoming Leader. State: Replica: 9f5b0087f2444520aa6722bbe2335bca, State: Running, Role: LEADER
I20260809 04:52:47.511444 19646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5b0087f2444520aa6722bbe2335bca" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41443 } }
I20260809 04:52:47.512044 19640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:52:47.514329 19651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f5b0087f2444520aa6722bbe2335bca. Latest consensus state: current_term: 1 leader_uuid: "9f5b0087f2444520aa6722bbe2335bca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5b0087f2444520aa6722bbe2335bca" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41443 } } }
I20260809 04:52:47.514791 19651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [sys.catalog]: This master's current role is: LEADER
I20260809 04:52:47.514719 19649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f5b0087f2444520aa6722bbe2335bca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5b0087f2444520aa6722bbe2335bca" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41443 } } }
I20260809 04:52:47.515004 19649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [sys.catalog]: This master's current role is: LEADER
I20260809 04:52:47.519891 19665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:52:47.524160 19665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:52:47.524838 19640 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 04:52:47.541894 19665 catalog_manager.cc:1347] Generated new cluster ID: 9e82438d5d7940208c79dcfed3364ce8
I20260809 04:52:47.542024 19665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:52:47.574379 19665 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:52:47.576306 19665 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:52:47.587605 19665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca: Generated new TSK 0
I20260809 04:52:47.588640 19665 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:52:48.348510 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.93.20:51503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:41443
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:52:48 dist-test-slave-4tdj krb5kdc[18859](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786251168, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 04:52:48.754994 19713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:52:48.755285 19713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:52:48.755455 19713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:52:48.762941 19713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:52:48.763062 19713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:52:48.763221 19713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:52:48.774765 19713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.93.20:51503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:41443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:52:48.776119 19713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:52:48.777488 19713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:52:48.796322 19725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:48.804055 19728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:48.804374 19726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:52:48.804805 19713 server_base.cc:1034] running on GCE node
I20260809 04:52:48.805790 19713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:52:48.814544 19713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:52:48.817857 19713 hybrid_clock.cc:648] HybridClock initialized: now 1786251168817776 us; error 63 us; skew 500 ppm
I20260809 04:52:48.818185 19713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:52:48 dist-test-slave-4tdj krb5kdc[18859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251168, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:52:48.821409 19713 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:52:48.823144 19713 webserver.cc:460] Webserver started at http://127.31.93.1:41193/ using document root <none> and password file <none>
I20260809 04:52:48.823851 19713 fs_manager.cc:359] Metadata directory not provided
I20260809 04:52:48.823968 19713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:52:48.824280 19713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 04:52:48 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251168, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:52:48.846828 19713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "77580d5658454f30b1f677b6808065a6"
format_stamp: "Formatted at 2026-08-09 04:52:48 on dist-test-slave-4tdj"
server_key: "4e2fede62f30bceb4bac1adcfc283df4"
server_key_iv: "9a8a21ba1f9682a28dc238ed15e735e9"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:48.847965 19713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "77580d5658454f30b1f677b6808065a6"
format_stamp: "Formatted at 2026-08-09 04:52:48 on dist-test-slave-4tdj"
server_key: "4e2fede62f30bceb4bac1adcfc283df4"
server_key_iv: "9a8a21ba1f9682a28dc238ed15e735e9"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:48.869515 19713 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.007s sys 0.012s
I20260809 04:52:48.894667 19740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:48.896159 19713 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260809 04:52:48.896334 19713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "77580d5658454f30b1f677b6808065a6"
format_stamp: "Formatted at 2026-08-09 04:52:48 on dist-test-slave-4tdj"
server_key: "4e2fede62f30bceb4bac1adcfc283df4"
server_key_iv: "9a8a21ba1f9682a28dc238ed15e735e9"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:48.896543 19713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:52:48.920756 19713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:52:48.925441 19713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:52:48.925771 19713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:52:48.927241 19713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:52:48.929005 19713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:52:48.929113 19713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:48.929236 19713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:52:48.929296 19713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:48.994756 19713 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:41931
I20260809 04:52:48.994843 19912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:41931 every 8 connection(s)
I20260809 04:52:48.996410 19713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:52:49.000504 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 19713
I20260809 04:52:49.000748 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:52:49.001799 32116 external_mini_cluster.cc:1428] Setting key 6405c7cc051a96c1618630f6d60217de
Aug 09 04:52:49 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251168, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
I20260809 04:52:49.054915 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:49.000032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41443 (local address 127.31.93.1:58253)
0809 04:52:49.004906 (+ 4874us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:49.004962 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:49.007715 (+ 2753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:49.019408 (+ 11693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:49.019421 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:49.020067 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:49.021016 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:49.021042 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:49.023989 (+ 2947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:49.023994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:49.025881 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:49.025897 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:49.029911 (+ 4014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:49.029919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:49.030241 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:49.031094 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:49.031129 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:49.034213 (+ 3084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:49.048718 (+ 14505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:49.048729 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:49.048748 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:49.049330 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:49.050988 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:49.050993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:49.050996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:49.051287 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:49.052206 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:49.052214 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:49.052638 (+ 424us) client_negotiation.cc:769] Sending connection context
0809 04:52:49.052943 (+ 305us) client_negotiation.cc:240] Negotiation successful
0809 04:52:49.053455 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4477,"thread_start_us":4602,"threads_started":1}
I20260809 04:52:49.057622 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:49.001867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:58253 (local address 127.31.93.62:41443)
0809 04:52:49.018669 (+ 16802us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:49.018675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:49.018695 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:52:49.018759 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:49.018763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:49.018850 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:49.021215 (+ 2365us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:49.021380 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:49.022497 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:49.026333 (+ 3836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:49.026884 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:49.032609 (+ 5725us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:49.034719 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:49.034724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:49.034728 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:49.034760 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:49.048531 (+ 13771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:49.050687 (+ 2156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:49.050693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:49.050697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:49.050851 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:49.051429 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:49.051433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:49.051436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:49.051787 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:49.056710 (+ 4923us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:49.056845 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 04:52:49.057105 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16669,"thread_start_us":17079,"threads_started":1}
I20260809 04:52:49.059661 19913 heartbeater.cc:347] Connected to a master server at 127.31.93.62:41443
I20260809 04:52:49.060003 19913 heartbeater.cc:464] Registering TS with master...
I20260809 04:52:49.060922 19913 heartbeater.cc:511] Master 127.31.93.62:41443 requested a full tablet report, sending...
I20260809 04:52:49.063753 19584 ts_manager.cc:194] Registered new tserver with Master: 77580d5658454f30b1f677b6808065a6 (127.31.93.1:41931)
I20260809 04:52:49.065726 19584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:58253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:52:49.075287 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.93.20:51503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:41443
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:52:49 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251168, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
Aug 09 04:52:49 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251168, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
W20260809 04:52:49.525458 19922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:52:49.525904 19922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:52:49.526101 19922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:52:49.536988 19922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:52:49.537186 19922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:52:49.537429 19922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:52:49.552868 19922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.93.20:51503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:41443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:52:49.554786 19922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:52:49.556536 19922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:52:49.597061 19933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:49.597922 19934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:49.599100 19941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:52:49.600754 19922 server_base.cc:1034] running on GCE node
I20260809 04:52:49.601811 19922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:52:49.609918 19922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:52:49.616245 19922 hybrid_clock.cc:648] HybridClock initialized: now 1786251169616071 us; error 84 us; skew 500 ppm
I20260809 04:52:49.616729 19922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:52:49 dist-test-slave-4tdj krb5kdc[18859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251169, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:52:49.625496 19922 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:52:49.649957 19922 webserver.cc:460] Webserver started at http://127.31.93.2:34909/ using document root <none> and password file <none>
I20260809 04:52:49.651139 19922 fs_manager.cc:359] Metadata directory not provided
I20260809 04:52:49.651361 19922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:52:49.651888 19922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 04:52:49 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251169, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:52:49.694350 19922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "366a686c052f49a2b3111928acab94bc"
format_stamp: "Formatted at 2026-08-09 04:52:49 on dist-test-slave-4tdj"
server_key: "ad625e843d3ed76325f5f8f750f1b58d"
server_key_iv: "0eae7a4c83a5ce07db944a6b0f5f9afa"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:49.699065 19922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "366a686c052f49a2b3111928acab94bc"
format_stamp: "Formatted at 2026-08-09 04:52:49 on dist-test-slave-4tdj"
server_key: "ad625e843d3ed76325f5f8f750f1b58d"
server_key_iv: "0eae7a4c83a5ce07db944a6b0f5f9afa"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:49.721342 19922 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.006s sys 0.002s
I20260809 04:52:49.764946 19951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:49.767149 19922 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.012s sys 0.004s
I20260809 04:52:49.767452 19922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "366a686c052f49a2b3111928acab94bc"
format_stamp: "Formatted at 2026-08-09 04:52:49 on dist-test-slave-4tdj"
server_key: "ad625e843d3ed76325f5f8f750f1b58d"
server_key_iv: "0eae7a4c83a5ce07db944a6b0f5f9afa"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:49.767798 19922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:52:49.815642 19922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:52:49.819518 19922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:52:49.819792 19922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:52:49.820812 19922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:52:49.822230 19922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:52:49.822338 19922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260809 04:52:49.822458 19922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:52:49.822513 19922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:49.860081 19922 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:40717
I20260809 04:52:49.860101 20129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:40717 every 8 connection(s)
I20260809 04:52:49.861423 19922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:52:49.865978 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 19922
I20260809 04:52:49.866137 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:52:49.866679 32116 external_mini_cluster.cc:1428] Setting key 874874ae1714fd490fdfd2dd7adb9fa7
Aug 09 04:52:49 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251169, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:52:49.881242 20132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:49.864132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:43231 (local address 127.31.93.62:41443)
0809 04:52:49.864621 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:49.864627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:49.866357 (+ 1730us) server_negotiation.cc:407] Connection header received
0809 04:52:49.868169 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:49.868174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:49.868245 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:49.868348 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:49.869740 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:49.870491 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:49.872038 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:49.872416 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:49.872526 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:49.875870 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:49.875875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:49.875878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:49.875910 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:49.878334 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:49.878971 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:49.878976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:49.878979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:49.879109 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:49.879523 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:49.879527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:49.879529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:49.879817 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:49.879948 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:49.880640 (+ 692us) server_negotiation.cc:299] Negotiation successful
0809 04:52:49.880866 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":220,"threads_started":1}
I20260809 04:52:49.882359 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:49.864658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41443 (local address 127.31.93.2:43231)
0809 04:52:49.865518 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:49.865576 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:49.867870 (+ 2294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:49.868590 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:49.868598 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:49.869021 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:49.869582 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:49.869600 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:49.870643 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:49.870647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:49.871906 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:49.871918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:49.872638 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:49.872646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:49.872946 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:49.873590 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:49.873616 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:49.875677 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:49.878456 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:49.878465 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:49.878480 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:49.878863 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:49.879210 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:49.879215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:49.879219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:49.879419 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:49.880008 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:49.880013 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:49.880297 (+ 284us) client_negotiation.cc:769] Sending connection context
0809 04:52:49.880535 (+ 238us) client_negotiation.cc:240] Negotiation successful
0809 04:52:49.881021 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":322,"threads_started":1}
I20260809 04:52:49.883986 20130 heartbeater.cc:347] Connected to a master server at 127.31.93.62:41443
I20260809 04:52:49.884289 20130 heartbeater.cc:464] Registering TS with master...
I20260809 04:52:49.885107 20130 heartbeater.cc:511] Master 127.31.93.62:41443 requested a full tablet report, sending...
I20260809 04:52:49.886816 19584 ts_manager.cc:194] Registered new tserver with Master: 366a686c052f49a2b3111928acab94bc (127.31.93.2:40717)
I20260809 04:52:49.887890 19584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:43231
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:52:49.927000 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.93.20:51503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:41443
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:52:50.069707 19913 heartbeater.cc:503] Master 127.31.93.62:41443 was elected leader, sending a full tablet report...
W20260809 04:52:50.256916 20137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:52:50.257189 20137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:52:50.257323 20137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:52:50.264480 20137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:52:50.264599 20137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:52:50.264752 20137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:52:50.274324 20137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.93.20:51503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:41443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:52:50.275574 20137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:52:50.276821 20137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:52:50.289187 20146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:50.289155 20145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:50.289340 20148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:52:50.290129 20137 server_base.cc:1034] running on GCE node
I20260809 04:52:50.290709 20137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:52:50.291596 20137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:52:50.292788 20137 hybrid_clock.cc:648] HybridClock initialized: now 1786251170292740 us; error 36 us; skew 500 ppm
I20260809 04:52:50.293080 20137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:52:50 dist-test-slave-4tdj krb5kdc[18859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251170, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:52:50.296026 20137 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:52:50.297340 20137 webserver.cc:460] Webserver started at http://127.31.93.3:44459/ using document root <none> and password file <none>
I20260809 04:52:50.297966 20137 fs_manager.cc:359] Metadata directory not provided
I20260809 04:52:50.298089 20137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:52:50.298419 20137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 04:52:50 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251170, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:52:50.313374 20137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "db3c597a764d4914b7217d782aa1e68b"
format_stamp: "Formatted at 2026-08-09 04:52:50 on dist-test-slave-4tdj"
server_key: "78baf5085b75a6d5c31dd57208a57194"
server_key_iv: "eab9535c20f7345ec5c5599bbd079203"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:50.314237 20137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "db3c597a764d4914b7217d782aa1e68b"
format_stamp: "Formatted at 2026-08-09 04:52:50 on dist-test-slave-4tdj"
server_key: "78baf5085b75a6d5c31dd57208a57194"
server_key_iv: "eab9535c20f7345ec5c5599bbd079203"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:50.319708 20137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 04:52:50.336017 20155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:50.337641 20137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 04:52:50.337814 20137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "db3c597a764d4914b7217d782aa1e68b"
format_stamp: "Formatted at 2026-08-09 04:52:50 on dist-test-slave-4tdj"
server_key: "78baf5085b75a6d5c31dd57208a57194"
server_key_iv: "eab9535c20f7345ec5c5599bbd079203"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:50.338058 20137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:52:50.371367 20137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:52:50.374894 20137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:52:50.375144 20137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:52:50.375978 20137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:52:50.377216 20137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:52:50.377287 20137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:50.377378 20137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:52:50.377419 20137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:50.407006 20137 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:46861
I20260809 04:52:50.407048 20322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:46861 every 8 connection(s)
I20260809 04:52:50.408169 20137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:52:50.412084 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 20137
I20260809 04:52:50.412292 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:52:50.412894 32116 external_mini_cluster.cc:1428] Setting key 5290df22715f8cffe937ff58228f5bbe
Aug 09 04:52:50 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251170, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:52:50.427136 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:50.410487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:55037 (local address 127.31.93.62:41443)
0809 04:52:50.410914 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:50.410918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:50.411913 (+ 995us) server_negotiation.cc:407] Connection header received
0809 04:52:50.414037 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:50.414043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:50.414147 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:50.414258 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:50.415807 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:50.416632 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:50.418071 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:50.418510 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:50.418605 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:50.421752 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:50.421756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:50.421759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:50.421797 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:50.424323 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:50.424970 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:50.424974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:50.424976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:50.425105 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:50.425492 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:50.425497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:50.425500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:50.425755 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:50.425905 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:50.426570 (+ 665us) server_negotiation.cc:299] Negotiation successful
0809 04:52:50.426777 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":205,"threads_started":1}
I20260809 04:52:50.428107 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:50.411038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41443 (local address 127.31.93.3:55037)
0809 04:52:50.411730 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:50.411786 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:50.413809 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:50.414532 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:50.414543 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:50.415089 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:50.415635 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:50.415653 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:50.416771 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:50.416775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:50.417956 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:50.417967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:50.418745 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:50.418751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:50.418961 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:50.419538 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:50.419561 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:50.421543 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:50.424448 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:50.424457 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:50.424471 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:50.424861 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:50.425206 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:50.425211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:50.425215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:50.425387 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:50.425876 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:50.425884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:50.426171 (+ 287us) client_negotiation.cc:769] Sending connection context
0809 04:52:50.426448 (+ 277us) client_negotiation.cc:240] Negotiation successful
0809 04:52:50.426898 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":286,"threads_started":1}
I20260809 04:52:50.429284 20324 heartbeater.cc:347] Connected to a master server at 127.31.93.62:41443
I20260809 04:52:50.429546 20324 heartbeater.cc:464] Registering TS with master...
I20260809 04:52:50.430173 20324 heartbeater.cc:511] Master 127.31.93.62:41443 requested a full tablet report, sending...
I20260809 04:52:50.431540 19584 ts_manager.cc:194] Registered new tserver with Master: db3c597a764d4914b7217d782aa1e68b (127.31.93.3:46861)
I20260809 04:52:50.432274 19584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:55037
I20260809 04:52:50.439726 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:52:50.441710 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 19713
Aug 09 04:52:50 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:52:50.457377 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:50.442547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.31.93.1:41931)
0809 04:52:50.443310 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:50.443316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:50.443339 (+ 23us) server_negotiation.cc:407] Connection header received
0809 04:52:50.443392 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:50.443399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:50.443624 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:50.443759 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:50.445060 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:50.446070 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:50.447472 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:50.447984 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:50.448086 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:50.452302 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:50.452314 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:50.452325 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:50.452360 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:50.455033 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:50.455671 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:50.455677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:50.455680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:50.455834 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:50.456109 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:50.456112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:50.456114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:50.456441 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:50.456692 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:50.456890 (+ 198us) server_negotiation.cc:299] Negotiation successful
0809 04:52:50.457093 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":405,"threads_started":1}
I20260809 04:52:50.458503 19906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:52:50.545598 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 19713
I20260809 04:52:50.567808 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 19922
Aug 09 04:52:50 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:52:50.583640 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:50.568750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33526 (local address 127.31.93.2:40717)
0809 04:52:50.569263 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:50.569270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:50.569295 (+ 25us) server_negotiation.cc:407] Connection header received
0809 04:52:50.569363 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:50.569370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:50.569562 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:50.569700 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:50.571013 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:50.572019 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:50.573460 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:50.573990 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:50.574088 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:50.578435 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:50.578445 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:50.578458 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:50.578491 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:50.580949 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:50.581603 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:50.581609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:50.581611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:50.581742 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:50.582102 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:50.582105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:50.582107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:50.582555 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:50.582785 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:50.583085 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 04:52:50.583275 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":260,"threads_started":1}
I20260809 04:52:50.584858 20124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:52:50.664876 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 19922
I20260809 04:52:50.685124 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 20137
Aug 09 04:52:50 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:52:50.700294 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:50.686125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.31.93.3:46861)
0809 04:52:50.686792 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:50.686799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:50.686827 (+ 28us) server_negotiation.cc:407] Connection header received
0809 04:52:50.686885 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:50.686891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:50.687092 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:50.687213 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:50.688476 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:50.689463 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:50.690755 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:50.691288 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:50.691380 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:50.695451 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:50.695461 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:50.695474 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:50.695507 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:50.697960 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:50.698695 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:50.698702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:50.698704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:50.698830 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:50.699111 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:50.699114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:50.699116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:50.699420 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:50.699651 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:50.699830 (+ 179us) server_negotiation.cc:299] Negotiation successful
0809 04:52:50.700023 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":347,"threads_started":1}
I20260809 04:52:50.701494 20317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:52:50.781894 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 20137
I20260809 04:52:50.804109 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 19541
I20260809 04:52:50.805524 19634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:52:50.908367 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 19541
I20260809 04:52:50.934000 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:41443
--webserver_interface=127.31.93.62
--webserver_port=34563
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:41443
--ranger_config_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.93.20:51503/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.93.20:51503/kms with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:52:51.251227 20340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:52:51.251500 20340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:52:51.251627 20340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:52:51.258574 20340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:52:51.258668 20340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:52:51.258741 20340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:52:51.258814 20340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:52:51.258878 20340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:52:51.268556 20340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.93.20:51503/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:41443
--ranger_config_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:41443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=34563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:52:51.269739 20340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:52:51.271037 20340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:52:51.282150 20351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:51.282240 20348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:51.282199 20349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:52:51.283102 20340 server_base.cc:1034] running on GCE node
I20260809 04:52:51.283823 20340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:52:51.285069 20340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:52:51.286266 20340 hybrid_clock.cc:648] HybridClock initialized: now 1786251171286214 us; error 47 us; skew 500 ppm
I20260809 04:52:51.286623 20340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:52:51 dist-test-slave-4tdj krb5kdc[18859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:52:51.289469 20340 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:52:51.290884 20340 webserver.cc:460] Webserver started at http://127.31.93.62:34563/ using document root <none> and password file <none>
I20260809 04:52:51.291492 20340 fs_manager.cc:359] Metadata directory not provided
I20260809 04:52:51.291618 20340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 04:52:51 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:52:51.339493 20340 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 04:52:51.342530 20360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:51.343896 20340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:52:51.344067 20340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "9f5b0087f2444520aa6722bbe2335bca"
format_stamp: "Formatted at 2026-08-09 04:52:47 on dist-test-slave-4tdj"
server_key: "78f06485fa3c8e75692a6fe5b9b10c1c"
server_key_iv: "b9b4013ed160722b4cb1f13c00acae96"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:51.344957 20340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:52:51.370199 20340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:52:51.373651 20340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:52:51.373893 20340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:52:51.390666 20340 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:41443
I20260809 04:52:51.390688 20436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:41443 every 8 connection(s)
I20260809 04:52:51.392022 20340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:52:51.396561 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 20340
I20260809 04:52:51.397716 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.93.20:51503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:41931
--local_ip_for_outbound_sockets=127.31.93.1
--tserver_master_addrs=127.31.93.62:41443
--webserver_port=41193
--webserver_interface=127.31.93.1
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.93.20:51503/kms with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:52:51.399442 20438 sys_catalog.cc:263] Verifying existing consensus state
I20260809 04:52:51.403741 20438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca: Bootstrap starting.
I20260809 04:52:51.422852 20438 log.cc:824] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca: Log is configured to *not* fsync() on all Append() calls
I20260809 04:52:51.431588 20438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:52:51.432070 20438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca: Bootstrap complete.
I20260809 04:52:51.440757 20438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5b0087f2444520aa6722bbe2335bca" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41443 } }
I20260809 04:52:51.441558 20438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f5b0087f2444520aa6722bbe2335bca, State: Initialized, Role: FOLLOWER
I20260809 04:52:51.442148 20438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5b0087f2444520aa6722bbe2335bca" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41443 } }
I20260809 04:52:51.442410 20438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:52:51.442536 20438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:52:51.442701 20438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 1 FOLLOWER]: Advancing to term 2
I20260809 04:52:51.446362 20438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5b0087f2444520aa6722bbe2335bca" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41443 } }
I20260809 04:52:51.446831 20438 leader_election.cc:302] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f5b0087f2444520aa6722bbe2335bca; no voters:
I20260809 04:52:51.447402 20438 leader_election.cc:288] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 04:52:51.447547 20446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 2 FOLLOWER]: Leader election won for term 2
I20260809 04:52:51.448657 20446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [term 2 LEADER]: Becoming Leader. State: Replica: 9f5b0087f2444520aa6722bbe2335bca, State: Running, Role: LEADER
I20260809 04:52:51.449258 20446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5b0087f2444520aa6722bbe2335bca" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41443 } }
I20260809 04:52:51.449719 20438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:52:51.451998 20448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9f5b0087f2444520aa6722bbe2335bca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5b0087f2444520aa6722bbe2335bca" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41443 } } }
I20260809 04:52:51.452378 20448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [sys.catalog]: This master's current role is: LEADER
I20260809 04:52:51.452831 20449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f5b0087f2444520aa6722bbe2335bca. Latest consensus state: current_term: 2 leader_uuid: "9f5b0087f2444520aa6722bbe2335bca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5b0087f2444520aa6722bbe2335bca" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41443 } } }
I20260809 04:52:51.453118 20449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca [sys.catalog]: This master's current role is: LEADER
I20260809 04:52:51.459506 20454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:52:51.465715 20454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:52:51.467263 20454 catalog_manager.cc:1259] Loaded cluster ID: 9e82438d5d7940208c79dcfed3364ce8
I20260809 04:52:51.467425 20454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:52:51.471645 20454 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:52:51.473201 20454 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca: Loaded TSK: 0
I20260809 04:52:51.475044 20454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:52:51.481454 20438 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260809 04:52:51.834165 20441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:52:51.834632 20441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:52:51.834827 20441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:52:51.846022 20441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:52:51.846165 20441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:52:51.846388 20441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:52:51.861382 20441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.93.20:51503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:41931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=41193
--tserver_master_addrs=127.31.93.62:41443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:52:51.863075 20441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:52:51.864915 20441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:52:51.926216 20500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:51.926817 20499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:51.927984 20504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:52:51.937769 20441 server_base.cc:1034] running on GCE node
I20260809 04:52:51.938673 20441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:52:51.944773 20441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:52:51.946702 20441 hybrid_clock.cc:648] HybridClock initialized: now 1786251171946620 us; error 77 us; skew 500 ppm
I20260809 04:52:51.947130 20441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:52:51 dist-test-slave-4tdj krb5kdc[18859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:52:51.951432 20441 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:52:51.959584 20441 webserver.cc:460] Webserver started at http://127.31.93.1:41193/ using document root <none> and password file <none>
I20260809 04:52:51.960644 20441 fs_manager.cc:359] Metadata directory not provided
I20260809 04:52:51.960861 20441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 04:52:51 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:52:51.991286 20441 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260809 04:52:51.996927 20518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:51.998883 20441 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.003s
I20260809 04:52:51.999146 20441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "77580d5658454f30b1f677b6808065a6"
format_stamp: "Formatted at 2026-08-09 04:52:48 on dist-test-slave-4tdj"
server_key: "4e2fede62f30bceb4bac1adcfc283df4"
server_key_iv: "9a8a21ba1f9682a28dc238ed15e735e9"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:52.000583 20441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:52:52.016717 20441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:52:52.020704 20441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:52:52.020954 20441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:52:52.021929 20441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:52:52.023326 20441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:52:52.023427 20441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:52.023540 20441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:52:52.023651 20441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 04:52:52.077219 20441 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:41931
I20260809 04:52:52.077270 20679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:41931 every 8 connection(s)
I20260809 04:52:52.078887 20441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:52:52.083550 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 20441
I20260809 04:52:52.084221 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.93.20:51503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:40717
--local_ip_for_outbound_sockets=127.31.93.2
--tserver_master_addrs=127.31.93.62:41443
--webserver_port=34909
--webserver_interface=127.31.93.2
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.93.20:51503/kms with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:52:52 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251171, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:52:52.112802 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:52.081967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41443 (local address 127.31.93.1:38935)
0809 04:52:52.082793 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:52.082848 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:52.088306 (+ 5458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:52.092312 (+ 4006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:52.092322 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:52.092853 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:52.093623 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:52.093646 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:52.096394 (+ 2748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:52.096399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:52.098219 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:52.098233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:52.099890 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:52.099898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:52.100165 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:52.100861 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:52.100889 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:52.103495 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:52.107710 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:52.107717 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:52.107730 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:52.108287 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:52.108842 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:52.108848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:52.108851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:52.109150 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:52.110117 (+ 967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:52.110125 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:52.110537 (+ 412us) client_negotiation.cc:769] Sending connection context
0809 04:52:52.110850 (+ 313us) client_negotiation.cc:240] Negotiation successful
0809 04:52:52.111391 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":356,"threads_started":1}
I20260809 04:52:52.112838 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:52.082061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:38935 (local address 127.31.93.62:41443)
0809 04:52:52.090774 (+ 8713us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:52.090792 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:52.090841 (+ 49us) server_negotiation.cc:407] Connection header received
0809 04:52:52.091241 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:52.091260 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:52.091762 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:52.093879 (+ 2117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:52.094341 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:52.096232 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:52.098685 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:52.099629 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:52.100599 (+ 970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:52.103785 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:52.103794 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:52.103814 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:52.103857 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:52.107562 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:52.108465 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:52.108474 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:52.108482 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:52.108692 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:52.109293 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:52.109298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:52.109301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:52.109949 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:52.110138 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:52.111121 (+ 983us) server_negotiation.cc:299] Negotiation successful
0809 04:52:52.111523 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8466,"thread_start_us":8229,"threads_started":1}
I20260809 04:52:52.116878 20682 heartbeater.cc:347] Connected to a master server at 127.31.93.62:41443
I20260809 04:52:52.117200 20682 heartbeater.cc:464] Registering TS with master...
I20260809 04:52:52.118105 20682 heartbeater.cc:511] Master 127.31.93.62:41443 requested a full tablet report, sending...
W20260809 04:52:52.121343 20682 heartbeater.cc:644] Failed to heartbeat to 127.31.93.62:41443 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 04:52:52.121565 20682 heartbeater.cc:464] Registering TS with master...
I20260809 04:52:52.121837 20682 heartbeater.cc:511] Master 127.31.93.62:41443 requested a full tablet report, sending...
I20260809 04:52:52.123103 20682 heartbeater.cc:464] Registering TS with master...
I20260809 04:52:52.123389 20682 heartbeater.cc:511] Master 127.31.93.62:41443 requested a full tablet report, sending...
W20260809 04:52:52.124454 20682 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 04:52:52.492214 20686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:52:52.492512 20686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:52:52.492674 20686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:52:52.502977 20686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:52:52.503201 20686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:52:52.503438 20686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:52:52.518814 20686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.93.20:51503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:40717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=34909
--tserver_master_addrs=127.31.93.62:41443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:52:52.520696 20686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:52:52.522524 20686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:52:52.541579 20712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:52.543195 20713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:52.562309 20715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:52:52.563066 20686 server_base.cc:1034] running on GCE node
I20260809 04:52:52.563758 20686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:52:52.564870 20686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:52:52.566131 20686 hybrid_clock.cc:648] HybridClock initialized: now 1786251172566044 us; error 78 us; skew 500 ppm
I20260809 04:52:52.566494 20686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:52:52 dist-test-slave-4tdj krb5kdc[18859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251172, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:52:52.569830 20686 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:52:52.572907 20686 webserver.cc:460] Webserver started at http://127.31.93.2:34909/ using document root <none> and password file <none>
I20260809 04:52:52.573635 20686 fs_manager.cc:359] Metadata directory not provided
I20260809 04:52:52.573770 20686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 04:52:52 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251172, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:52:52.596141 20686 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 04:52:52.599831 20727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:52.601406 20686 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:52:52.601576 20686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "366a686c052f49a2b3111928acab94bc"
format_stamp: "Formatted at 2026-08-09 04:52:49 on dist-test-slave-4tdj"
server_key: "ad625e843d3ed76325f5f8f750f1b58d"
server_key_iv: "0eae7a4c83a5ce07db944a6b0f5f9afa"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:52.602809 20686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 09 04:52:52 dist-test-slave-4tdj krb5kdc[18859](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786251172, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:52:52.639230 20686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:52:52.644516 20686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:52:52.644883 20686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:52:52.647421 20686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:52:52.652976 20686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:52:52.653084 20686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:52.653198 20686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:52:52.653267 20686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:52.704645 20686 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:40717
I20260809 04:52:52.704725 20903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:40717 every 8 connection(s)
I20260809 04:52:52.706660 20686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:52:52.713495 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 20686
I20260809 04:52:52.714339 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.93.20:51503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:46861
--local_ip_for_outbound_sockets=127.31.93.3
--tserver_master_addrs=127.31.93.62:41443
--webserver_port=44459
--webserver_interface=127.31.93.3
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.93.20:51503/kms with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:52:52 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251172, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:52:52.754704 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:52.709692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:34005 (local address 127.31.93.62:41443)
0809 04:52:52.737313 (+ 27621us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:52.737320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:52.737341 (+ 21us) server_negotiation.cc:407] Connection header received
0809 04:52:52.737407 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:52.737413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:52.737498 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:52.737625 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:52.739879 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:52.741087 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:52.744097 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:52.744756 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:52.744854 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:52.748596 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:52.748603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:52.748607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:52.748647 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:52.751377 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:52.752045 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:52.752051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:52.752055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:52.752176 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:52.752573 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:52.752578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:52.752581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:52.752869 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:52.752998 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:52.754236 (+ 1238us) server_negotiation.cc:299] Negotiation successful
0809 04:52:52.754439 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27479,"thread_start_us":32122,"threads_started":1}
I20260809 04:52:52.756377 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:52.710441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41443 (local address 127.31.93.2:34005)
0809 04:52:52.732908 (+ 22467us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:52.732976 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:52.736595 (+ 3619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:52.738071 (+ 1476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:52.738085 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:52.738797 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:52.739659 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:52.739685 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:52.741877 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:52.741882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:52.743923 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:52.743938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:52.745187 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:52.745195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:52.745495 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:52.746336 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:52.746377 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:52.748376 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:52.751497 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:52.751514 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:52.751530 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:52.751916 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:52.752277 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:52.752282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:52.752286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:52.752461 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:52.752984 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:52.752992 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:52.753378 (+ 386us) client_negotiation.cc:769] Sending connection context
0809 04:52:52.754957 (+ 1579us) client_negotiation.cc:240] Negotiation successful
0809 04:52:52.755375 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22072,"thread_start_us":23738,"threads_started":1}
I20260809 04:52:52.757808 20904 heartbeater.cc:347] Connected to a master server at 127.31.93.62:41443
I20260809 04:52:52.758095 20904 heartbeater.cc:464] Registering TS with master...
I20260809 04:52:52.758826 20904 heartbeater.cc:511] Master 127.31.93.62:41443 requested a full tablet report, sending...
I20260809 04:52:52.760506 20385 ts_manager.cc:194] Registered new tserver with Master: 366a686c052f49a2b3111928acab94bc (127.31.93.2:40717)
I20260809 04:52:52.762120 20385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:34005
Aug 09 04:52:53 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251172, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:52:53.124601 20682 heartbeater.cc:464] Registering TS with master...
I20260809 04:52:53.125043 20682 heartbeater.cc:511] Master 127.31.93.62:41443 requested a full tablet report, sending...
I20260809 04:52:53.126341 20385 ts_manager.cc:194] Registered new tserver with Master: 77580d5658454f30b1f677b6808065a6 (127.31.93.1:41931)
I20260809 04:52:53.127468 20385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:38935
W20260809 04:52:53.186902 20914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:52:53.187203 20914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:52:53.187390 20914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:52:53.194700 20914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:52:53.194821 20914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:52:53.195001 20914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:52:53.204905 20914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.93.20:51503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:46861
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=44459
--tserver_master_addrs=127.31.93.62:41443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:52:53.206558 20914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:52:53.208016 20914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:52:53.224807 20922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:53.224849 20921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:53.232162 20924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:52:53.232645 20914 server_base.cc:1034] running on GCE node
I20260809 04:52:53.233599 20914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:52:53.234953 20914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:52:53.236290 20914 hybrid_clock.cc:648] HybridClock initialized: now 1786251173236209 us; error 58 us; skew 500 ppm
I20260809 04:52:53.236750 20914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:52:53 dist-test-slave-4tdj krb5kdc[18859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251173, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:52:53.241182 20914 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:52:53.243255 20914 webserver.cc:460] Webserver started at http://127.31.93.3:44459/ using document root <none> and password file <none>
I20260809 04:52:53.244246 20914 fs_manager.cc:359] Metadata directory not provided
I20260809 04:52:53.244477 20914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 04:52:53 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251173, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:52:53.273723 20914 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.007s sys 0.000s
I20260809 04:52:53.277379 20937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:53.279208 20914 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260809 04:52:53.279366 20914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "db3c597a764d4914b7217d782aa1e68b"
format_stamp: "Formatted at 2026-08-09 04:52:50 on dist-test-slave-4tdj"
server_key: "78baf5085b75a6d5c31dd57208a57194"
server_key_iv: "eab9535c20f7345ec5c5599bbd079203"
server_key_version: "kuduclusterkey@0"
I20260809 04:52:53.280646 20914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:52:53.322469 20914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:52:53.328707 20914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:52:53.329000 20914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:52:53.330727 20914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:52:53.332546 20914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:52:53.332710 20914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260809 04:52:53.332824 20914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:52:53.332906 20914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 04:52:53 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251172, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for HTTP/127.31.93.20@KRBTEST.COM
I20260809 04:52:53.765676 20904 heartbeater.cc:503] Master 127.31.93.62:41443 was elected leader, sending a full tablet report...
I20260809 04:52:53.789014 20914 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:46861
I20260809 04:52:53.789047 21113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:46861 every 8 connection(s)
I20260809 04:52:53.790607 20914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:52:53.791088 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 20914
I20260809 04:52:53.811861 21117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:53.792248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41348 (local address 127.31.93.62:41443)
0809 04:52:53.794567 (+ 2319us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:53.794573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:53.794594 (+ 21us) server_negotiation.cc:407] Connection header received
0809 04:52:53.794673 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:53.794679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:53.794786 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:53.794902 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:53.797167 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.798449 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.800616 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.801369 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.801484 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:53.805500 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:53.805509 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:53.805515 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:53.805561 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:53.809210 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:53.809808 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:53.809813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:53.809815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:53.809941 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:53.810309 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:53.810315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:53.810318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:53.810700 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:53.810922 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:53.811288 (+ 366us) server_negotiation.cc:299] Negotiation successful
0809 04:52:53.811489 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2197,"thread_start_us":2349,"threads_started":1}
Aug 09 04:52:53 dist-test-slave-4tdj krb5kdc[18859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251173, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:52:53.826200 21119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:53.794488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:44997 (local address 127.31.93.62:41443)
0809 04:52:53.808587 (+ 14099us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:53.808592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:53.808612 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:52:53.810444 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:53.810449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:53.810543 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:53.810648 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:53.812587 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.813755 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.816066 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.816543 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.816633 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:53.820318 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:53.820323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:53.820326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:53.820355 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:53.822694 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:53.823470 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:53.823473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:53.823475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:53.823577 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:53.823968 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:53.823971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:53.823973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:53.824236 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:53.824372 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:53.825656 (+ 1284us) server_negotiation.cc:299] Negotiation successful
0809 04:52:53.825851 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13963,"thread_start_us":13842,"threads_started":1}
I20260809 04:52:53.826784 21120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:53.794355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41443 (local address 127.31.93.3:44997)
0809 04:52:53.807683 (+ 13328us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:53.807750 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:53.810140 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:53.810899 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:53.810911 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:53.811505 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:53.812327 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:53.812364 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.813941 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.813946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:53.815932 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:53.815948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.816755 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.816763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:53.817024 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:53.817648 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:53.817677 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:53.820097 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:53.822814 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:53.822823 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:53.822838 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:53.823357 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:53.823658 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:53.823662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:53.823665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:53.823881 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:53.824350 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:53.824358 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:53.824739 (+ 381us) client_negotiation.cc:769] Sending connection context
0809 04:52:53.825002 (+ 263us) client_negotiation.cc:240] Negotiation successful
0809 04:52:53.825476 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12959,"thread_start_us":12806,"threads_started":1}
I20260809 04:52:53.828029 21114 heartbeater.cc:347] Connected to a master server at 127.31.93.62:41443
I20260809 04:52:53.828317 21114 heartbeater.cc:464] Registering TS with master...
I20260809 04:52:53.829015 21114 heartbeater.cc:511] Master 127.31.93.62:41443 requested a full tablet report, sending...
I20260809 04:52:53.830353 20385 ts_manager.cc:194] Registered new tserver with Master: db3c597a764d4914b7217d782aa1e68b (127.31.93.3:46861)
I20260809 04:52:53.831127 20385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:44997
I20260809 04:52:53.838173 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:52:53.854563 21119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:53.841276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.31.93.62:41443)
0809 04:52:53.841466 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:53.841472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:53.841937 (+ 465us) server_negotiation.cc:407] Connection header received
0809 04:52:53.842066 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:53.842071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:53.842166 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:53.842271 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:53.843708 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.844841 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.846160 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.846836 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.846966 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:53.849513 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:53.849520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:53.849524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:53.849563 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:53.852124 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:53.852737 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:53.852742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:53.852745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:53.852860 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:53.853209 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:53.853215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:53.853218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:53.853485 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:53.853708 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:53.854024 (+ 316us) server_negotiation.cc:299] Negotiation successful
0809 04:52:53.854233 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 04:52:53.859218 20385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:52:53.866578 20385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:52:53.900283 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:53.880165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:41931 (local address 127.0.0.1:54232)
0809 04:52:53.891402 (+ 11237us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:53.891443 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:53.891645 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:53.892645 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:53.892653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:53.892682 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:52:53.893066 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:53.893078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.895011 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.895016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:53.897294 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:53.897310 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.899394 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.899400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:53.899599 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:53.899647 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 04:52:53.899773 (+ 126us) client_negotiation.cc:240] Negotiation successful
0809 04:52:53.899936 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10972,"threads_started":1}
I20260809 04:52:53.906234 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:53.880395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.31.93.1:41931)
0809 04:52:53.889608 (+ 9213us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:53.889618 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:53.891831 (+ 2213us) server_negotiation.cc:407] Connection header received
0809 04:52:53.891936 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:53.891944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:53.892177 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:53.892354 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:52:53.893278 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.894869 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.897494 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.898906 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.899019 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:53.899229 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:53.905745 (+ 6516us) server_negotiation.cc:299] Negotiation successful
0809 04:52:53.905908 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9022,"thread_start_us":8836,"threads_started":1}
I20260809 04:52:53.910391 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:53.881329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.31.93.2:40717)
0809 04:52:53.891012 (+ 9683us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:53.891020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:53.900870 (+ 9850us) server_negotiation.cc:407] Connection header received
0809 04:52:53.901129 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:53.901137 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:53.901399 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:53.901590 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:52:53.902334 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.903934 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.907516 (+ 3582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.909012 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.909116 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:53.909341 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:53.909863 (+ 522us) server_negotiation.cc:299] Negotiation successful
0809 04:52:53.910025 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9507,"thread_start_us":9322,"threads_started":1}
I20260809 04:52:53.911203 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:53.880283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:40717 (local address 127.0.0.1:33540)
0809 04:52:53.900757 (+ 20474us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:53.900774 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:53.900882 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:53.901893 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:53.901898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:53.901925 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:52:53.902114 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:53.902125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.904100 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.904105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:53.906842 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:53.906857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.909490 (+ 2633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.909495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:53.909675 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:53.909703 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 04:52:53.910791 (+ 1088us) client_negotiation.cc:240] Negotiation successful
0809 04:52:53.910945 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20368,"thread_start_us":10727,"threads_started":1}
I20260809 04:52:53.911863 20587 tablet_service.cc:1467] Processing CreateTablet for tablet d9393cb1953141fdbb39a37d5d31bf87 (DEFAULT_TABLE table=test-table [id=7e1b3a834422480b99be926bfbba9484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:52:53.913885 20587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9393cb1953141fdbb39a37d5d31bf87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:52:53.914821 20805 tablet_service.cc:1467] Processing CreateTablet for tablet d9393cb1953141fdbb39a37d5d31bf87 (DEFAULT_TABLE table=test-table [id=7e1b3a834422480b99be926bfbba9484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:52:53.915812 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:53.898170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:46861 (local address 127.0.0.1:59280)
0809 04:52:53.905746 (+ 7576us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:53.905770 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:53.905886 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:53.908754 (+ 2868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:53.908758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:53.908784 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:52:53.909147 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:53.909159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.911198 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.911203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:53.913058 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:53.913073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.915099 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.915106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:53.915261 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:53.915292 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 04:52:53.915376 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 04:52:53.915503 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7439,"thread_start_us":7784,"threads_started":1}
I20260809 04:52:53.916427 20805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9393cb1953141fdbb39a37d5d31bf87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:52:53.916502 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:53.892357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.31.93.3:46861)
0809 04:52:53.907908 (+ 15551us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:53.907917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:53.907950 (+ 33us) server_negotiation.cc:407] Connection header received
0809 04:52:53.908041 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:53.908048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:53.908275 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:53.908444 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:52:53.909485 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.911034 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.913237 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:53.914593 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:53.914730 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:53.914948 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:53.916029 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0809 04:52:53.916181 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15377,"thread_start_us":16882,"threads_started":1}
I20260809 04:52:53.920624 21008 tablet_service.cc:1467] Processing CreateTablet for tablet d9393cb1953141fdbb39a37d5d31bf87 (DEFAULT_TABLE table=test-table [id=7e1b3a834422480b99be926bfbba9484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:52:53.922044 21008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9393cb1953141fdbb39a37d5d31bf87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:52:53.937220 21149 tablet_bootstrap.cc:492] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc: Bootstrap starting.
I20260809 04:52:53.939754 21147 tablet_bootstrap.cc:492] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6: Bootstrap starting.
I20260809 04:52:53.942924 21150 tablet_bootstrap.cc:492] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b: Bootstrap starting.
I20260809 04:52:53.943039 21147 tablet_bootstrap.cc:654] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6: Neither blocks nor log segments found. Creating new log.
I20260809 04:52:53.943192 21149 tablet_bootstrap.cc:654] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc: Neither blocks nor log segments found. Creating new log.
I20260809 04:52:53.944329 21149 log.cc:824] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc: Log is configured to *not* fsync() on all Append() calls
I20260809 04:52:53.944330 21147 log.cc:824] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6: Log is configured to *not* fsync() on all Append() calls
I20260809 04:52:53.947283 21150 tablet_bootstrap.cc:654] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b: Neither blocks nor log segments found. Creating new log.
I20260809 04:52:53.949524 21150 log.cc:824] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b: Log is configured to *not* fsync() on all Append() calls
I20260809 04:52:53.954679 21147 tablet_bootstrap.cc:492] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6: No bootstrap required, opened a new log
I20260809 04:52:53.954880 21149 tablet_bootstrap.cc:492] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc: No bootstrap required, opened a new log
I20260809 04:52:53.954945 21147 ts_tablet_manager.cc:1397] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260809 04:52:53.955247 21150 tablet_bootstrap.cc:492] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b: No bootstrap required, opened a new log
I20260809 04:52:53.955241 21149 ts_tablet_manager.cc:1397] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.007s
I20260809 04:52:53.955498 21150 ts_tablet_manager.cc:1397] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.007s
I20260809 04:52:53.965587 21149 raft_consensus.cc:348] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77580d5658454f30b1f677b6808065a6" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41931 } } peers { permanent_uuid: "366a686c052f49a2b3111928acab94bc" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40717 } } peers { permanent_uuid: "db3c597a764d4914b7217d782aa1e68b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46861 } }
I20260809 04:52:53.965580 21150 raft_consensus.cc:348] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77580d5658454f30b1f677b6808065a6" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41931 } } peers { permanent_uuid: "366a686c052f49a2b3111928acab94bc" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40717 } } peers { permanent_uuid: "db3c597a764d4914b7217d782aa1e68b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46861 } }
I20260809 04:52:53.966001 21150 raft_consensus.cc:374] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:52:53.966104 21150 raft_consensus.cc:729] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3c597a764d4914b7217d782aa1e68b, State: Initialized, Role: FOLLOWER
I20260809 04:52:53.965924 21147 raft_consensus.cc:348] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77580d5658454f30b1f677b6808065a6" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41931 } } peers { permanent_uuid: "366a686c052f49a2b3111928acab94bc" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40717 } } peers { permanent_uuid: "db3c597a764d4914b7217d782aa1e68b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46861 } }
I20260809 04:52:53.966396 21147 raft_consensus.cc:374] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:52:53.966434 21149 raft_consensus.cc:374] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:52:53.966514 21147 raft_consensus.cc:729] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77580d5658454f30b1f677b6808065a6, State: Initialized, Role: FOLLOWER
I20260809 04:52:53.966640 21149 raft_consensus.cc:729] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 366a686c052f49a2b3111928acab94bc, State: Initialized, Role: FOLLOWER
I20260809 04:52:53.966830 21150 consensus_queue.cc:260] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77580d5658454f30b1f677b6808065a6" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41931 } } peers { permanent_uuid: "366a686c052f49a2b3111928acab94bc" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40717 } } peers { permanent_uuid: "db3c597a764d4914b7217d782aa1e68b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46861 } }
I20260809 04:52:53.967301 21147 consensus_queue.cc:260] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77580d5658454f30b1f677b6808065a6" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41931 } } peers { permanent_uuid: "366a686c052f49a2b3111928acab94bc" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40717 } } peers { permanent_uuid: "db3c597a764d4914b7217d782aa1e68b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46861 } }
I20260809 04:52:53.967514 21149 consensus_queue.cc:260] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77580d5658454f30b1f677b6808065a6" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41931 } } peers { permanent_uuid: "366a686c052f49a2b3111928acab94bc" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40717 } } peers { permanent_uuid: "db3c597a764d4914b7217d782aa1e68b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46861 } }
I20260809 04:52:53.973646 21149 ts_tablet_manager.cc:1428] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc: Time spent starting tablet: real 0.018s user 0.010s sys 0.005s
I20260809 04:52:53.974828 21150 ts_tablet_manager.cc:1428] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b: Time spent starting tablet: real 0.019s user 0.015s sys 0.004s
I20260809 04:52:53.975162 21114 heartbeater.cc:503] Master 127.31.93.62:41443 was elected leader, sending a full tablet report...
I20260809 04:52:53.978017 20682 heartbeater.cc:503] Master 127.31.93.62:41443 was elected leader, sending a full tablet report...
I20260809 04:52:53.978587 21147 ts_tablet_manager.cc:1428] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6: Time spent starting tablet: real 0.024s user 0.017s sys 0.005s
W20260809 04:52:53.992767 20905 tablet.cc:2369] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:52:54.044605 21115 tablet.cc:2369] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:52:54.062712 21155 raft_consensus.cc:482] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:52:54.063014 21155 raft_consensus.cc:504] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77580d5658454f30b1f677b6808065a6" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41931 } } peers { permanent_uuid: "366a686c052f49a2b3111928acab94bc" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40717 } } peers { permanent_uuid: "db3c597a764d4914b7217d782aa1e68b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46861 } }
I20260809 04:52:54.065002 21155 leader_election.cc:288] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 366a686c052f49a2b3111928acab94bc (127.31.93.2:40717), db3c597a764d4914b7217d782aa1e68b (127.31.93.3:46861)
I20260809 04:52:54.072590 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:54.065387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:43985 (local address 127.31.93.2:40717)
0809 04:52:54.065588 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:54.065594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:54.066201 (+ 607us) server_negotiation.cc:407] Connection header received
0809 04:52:54.066548 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:54.066555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:54.066658 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:54.066725 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:52:54.067606 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:54.068704 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:54.070875 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:54.071715 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:54.071827 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:54.071934 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:54.072230 (+ 296us) server_negotiation.cc:299] Negotiation successful
0809 04:52:54.072356 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 04:52:54.072702 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:54.065215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:40717 (local address 127.31.93.1:43985)
0809 04:52:54.066102 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:54.066121 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:54.066359 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:54.066943 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:54.066948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:54.066979 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:52:54.067431 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:54.067445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:54.068850 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:54.068855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:54.070707 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:54.070720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:54.071932 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:54.071938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:54.072080 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:54.072111 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 04:52:54.072200 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 04:52:54.072327 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":560,"threads_started":1}
I20260809 04:52:54.073679 20841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9393cb1953141fdbb39a37d5d31bf87" candidate_uuid: "77580d5658454f30b1f677b6808065a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366a686c052f49a2b3111928acab94bc" is_pre_election: true
I20260809 04:52:54.074270 20841 raft_consensus.cc:2454] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77580d5658454f30b1f677b6808065a6 in term 0.
I20260809 04:52:54.075052 20521 leader_election.cc:302] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366a686c052f49a2b3111928acab94bc, 77580d5658454f30b1f677b6808065a6; no voters:
I20260809 04:52:54.075461 21155 raft_consensus.cc:2781] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:52:54.075618 21155 raft_consensus.cc:482] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:52:54.075757 21155 raft_consensus.cc:3037] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:52:54.075876 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:54.066221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:46861 (local address 127.31.93.1:37517)
0809 04:52:54.067991 (+ 1770us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:54.068005 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:54.068118 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:54.068394 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:54.068398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:54.068419 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:52:54.068721 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:54.068730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:54.070850 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:54.070856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:54.073273 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:54.073288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:54.075005 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:54.075013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:54.075198 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:54.075231 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 04:52:54.075363 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 04:52:54.075489 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1675,"thread_start_us":1494,"threads_started":1}
I20260809 04:52:54.076110 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:54.066369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:37517 (local address 127.31.93.3:46861)
0809 04:52:54.066548 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:54.066553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:54.068102 (+ 1549us) server_negotiation.cc:407] Connection header received
0809 04:52:54.068236 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:54.068240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:54.068298 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:54.068371 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:52:54.069639 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:54.070695 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:54.073451 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:54.074639 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:54.074756 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:54.074852 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:54.075415 (+ 563us) server_negotiation.cc:299] Negotiation successful
0809 04:52:54.075617 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:52:54.078255 21155 raft_consensus.cc:504] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77580d5658454f30b1f677b6808065a6" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41931 } } peers { permanent_uuid: "366a686c052f49a2b3111928acab94bc" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40717 } } peers { permanent_uuid: "db3c597a764d4914b7217d782aa1e68b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46861 } }
I20260809 04:52:54.078975 21155 leader_election.cc:288] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [CANDIDATE]: Term 1 election: Requested vote from peers 366a686c052f49a2b3111928acab94bc (127.31.93.2:40717), db3c597a764d4914b7217d782aa1e68b (127.31.93.3:46861)
I20260809 04:52:54.079136 21036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9393cb1953141fdbb39a37d5d31bf87" candidate_uuid: "77580d5658454f30b1f677b6808065a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3c597a764d4914b7217d782aa1e68b" is_pre_election: true
I20260809 04:52:54.079326 20841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9393cb1953141fdbb39a37d5d31bf87" candidate_uuid: "77580d5658454f30b1f677b6808065a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366a686c052f49a2b3111928acab94bc"
I20260809 04:52:54.079519 20841 raft_consensus.cc:3037] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:52:54.079695 21036 raft_consensus.cc:2454] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77580d5658454f30b1f677b6808065a6 in term 0.
W20260809 04:52:54.081790 20683 tablet.cc:2369] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:52:54.082448 20841 raft_consensus.cc:2454] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77580d5658454f30b1f677b6808065a6 in term 1.
I20260809 04:52:54.082896 20521 leader_election.cc:302] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366a686c052f49a2b3111928acab94bc, 77580d5658454f30b1f677b6808065a6; no voters:
I20260809 04:52:54.082851 21033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9393cb1953141fdbb39a37d5d31bf87" candidate_uuid: "77580d5658454f30b1f677b6808065a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db3c597a764d4914b7217d782aa1e68b"
I20260809 04:52:54.083104 21033 raft_consensus.cc:3037] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:52:54.085639 21033 raft_consensus.cc:2454] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77580d5658454f30b1f677b6808065a6 in term 1.
I20260809 04:52:54.085919 21155 raft_consensus.cc:2781] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:52:54.086438 21155 raft_consensus.cc:686] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [term 1 LEADER]: Becoming Leader. State: Replica: 77580d5658454f30b1f677b6808065a6, State: Running, Role: LEADER
I20260809 04:52:54.086912 21155 consensus_queue.cc:237] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77580d5658454f30b1f677b6808065a6" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41931 } } peers { permanent_uuid: "366a686c052f49a2b3111928acab94bc" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40717 } } peers { permanent_uuid: "db3c597a764d4914b7217d782aa1e68b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46861 } }
I20260809 04:52:54.091217 20380 catalog_manager.cc:5495] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77580d5658454f30b1f677b6808065a6 (127.31.93.1). New cstate: current_term: 1 leader_uuid: "77580d5658454f30b1f677b6808065a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77580d5658454f30b1f677b6808065a6" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "366a686c052f49a2b3111928acab94bc" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db3c597a764d4914b7217d782aa1e68b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46861 } health_report { overall_health: UNKNOWN } } }
I20260809 04:52:54.115742 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:54.109774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54234 (local address 127.31.93.1:41931)
0809 04:52:54.109928 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:54.109932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:54.110185 (+ 253us) server_negotiation.cc:407] Connection header received
0809 04:52:54.110423 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:54.110426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:54.110513 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:54.110632 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:52:54.111028 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:54.111726 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:54.113656 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:54.114024 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:54.114118 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:54.114714 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:52:54.115211 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:52:54.115318 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:54.115430 (+ 112us) server_negotiation.cc:299] Negotiation successful
0809 04:52:54.115495 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 04:52:54.123490 21033 raft_consensus.cc:1261] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b [term 1 FOLLOWER]: Refusing update from remote peer 77580d5658454f30b1f677b6808065a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:52:54.123526 20841 raft_consensus.cc:1261] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc [term 1 FOLLOWER]: Refusing update from remote peer 77580d5658454f30b1f677b6808065a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:52:54.124341 21161 consensus_queue.cc:1035] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db3c597a764d4914b7217d782aa1e68b" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 46861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:52:54.124733 21155 consensus_queue.cc:1035] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "366a686c052f49a2b3111928acab94bc" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:52:54.140342 21172 mvcc.cc:205] Tried to move back new op lower bound from 7316484809200603136 to 7316484809063788544. Current Snapshot: MvccSnapshot[applied={T|T < 7316484809200603136}]
I20260809 04:52:54.149957 20385 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:52:54.150270 20385 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:52:54.153703 20385 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f5b0087f2444520aa6722bbe2335bca: Sending DeleteTablet for 3 replicas of tablet d9393cb1953141fdbb39a37d5d31bf87
I20260809 04:52:54.154763 20805 tablet_service.cc:1514] Processing DeleteTablet for tablet d9393cb1953141fdbb39a37d5d31bf87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:52:54 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:33540
I20260809 04:52:54.154776 20587 tablet_service.cc:1514] Processing DeleteTablet for tablet d9393cb1953141fdbb39a37d5d31bf87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:52:54 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:54232
I20260809 04:52:54.155646 21184 tablet_replica.cc:330] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6: stopping tablet replica
I20260809 04:52:54.155953 21008 tablet_service.cc:1514] Processing DeleteTablet for tablet d9393cb1953141fdbb39a37d5d31bf87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:52:54 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:59280
I20260809 04:52:54.156091 21184 raft_consensus.cc:2229] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:52:54.156813 21184 raft_consensus.cc:2258] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:52:54.158854 21184 ts_tablet_manager.cc:1905] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:52:54.161827 21188 tablet_replica.cc:330] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b: stopping tablet replica
I20260809 04:52:54.162401 21188 raft_consensus.cc:2229] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:52:54.162920 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 20441
I20260809 04:52:54.163144 21188 raft_consensus.cc:2258] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:52:54.166320 21184 ts_tablet_manager.cc:1918] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:52:54.166548 21184 log.cc:1196] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/d9393cb1953141fdbb39a37d5d31bf87
I20260809 04:52:54.167166 21184 ts_tablet_manager.cc:1939] T d9393cb1953141fdbb39a37d5d31bf87 P 77580d5658454f30b1f677b6808065a6: Deleting consensus metadata
I20260809 04:52:54.167142 21188 ts_tablet_manager.cc:1905] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:52:54.167972 21187 tablet_replica.cc:330] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc: stopping tablet replica
I20260809 04:52:54.168494 21187 raft_consensus.cc:2229] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:52:54.169145 21187 raft_consensus.cc:2258] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:52:54.169272 20364 catalog_manager.cc:4841] TS 77580d5658454f30b1f677b6808065a6 (127.31.93.1:41931): tablet d9393cb1953141fdbb39a37d5d31bf87 (table test-table [id=7e1b3a834422480b99be926bfbba9484]) successfully deleted
I20260809 04:52:54.171137 21187 ts_tablet_manager.cc:1905] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:52:54.174081 21188 ts_tablet_manager.cc:1918] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:52:54.174237 21188 log.cc:1196] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/d9393cb1953141fdbb39a37d5d31bf87
I20260809 04:52:54.174772 21188 ts_tablet_manager.cc:1939] T d9393cb1953141fdbb39a37d5d31bf87 P db3c597a764d4914b7217d782aa1e68b: Deleting consensus metadata
I20260809 04:52:54.176504 20363 catalog_manager.cc:4841] TS db3c597a764d4914b7217d782aa1e68b (127.31.93.3:46861): tablet d9393cb1953141fdbb39a37d5d31bf87 (table test-table [id=7e1b3a834422480b99be926bfbba9484]) successfully deleted
I20260809 04:52:54.177531 21187 ts_tablet_manager.cc:1918] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:52:54.177686 21187 log.cc:1196] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/d9393cb1953141fdbb39a37d5d31bf87
I20260809 04:52:54.178089 21187 ts_tablet_manager.cc:1939] T d9393cb1953141fdbb39a37d5d31bf87 P 366a686c052f49a2b3111928acab94bc: Deleting consensus metadata
I20260809 04:52:54.179641 20363 catalog_manager.cc:4841] TS 366a686c052f49a2b3111928acab94bc (127.31.93.2:40717): tablet d9393cb1953141fdbb39a37d5d31bf87 (table test-table [id=7e1b3a834422480b99be926bfbba9484]) successfully deleted
I20260809 04:52:54.185017 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:54.164171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54242 (local address 127.31.93.1:41931)
0809 04:52:54.164356 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:54.164362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:54.164384 (+ 22us) server_negotiation.cc:407] Connection header received
0809 04:52:54.164626 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:54.164632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:54.164727 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:54.164835 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:54.167894 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:54.168990 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:54.170695 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:54.171123 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:54.171252 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:54.178435 (+ 7183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:54.178448 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:54.178465 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:54.178517 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:54.181994 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:54.182828 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:54.182835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:54.182838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:54.183002 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:54.183445 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:54.183449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:54.183451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:54.183803 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:54.184029 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:54.184455 (+ 426us) server_negotiation.cc:299] Negotiation successful
0809 04:52:54.184685 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 04:52:54.185676 20674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:52:54.279362 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 20441
I20260809 04:52:54.303376 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 20686
I20260809 04:52:54.320624 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:54.304170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33556 (local address 127.31.93.2:40717)
0809 04:52:54.304335 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:54.304339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:54.304410 (+ 71us) server_negotiation.cc:407] Connection header received
0809 04:52:54.304576 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:54.304580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:54.304646 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:54.304728 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:54.306020 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:54.306856 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:54.308269 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:54.308693 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:54.308784 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:54.315358 (+ 6574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:54.315372 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:54.315387 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:54.315421 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:54.318190 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:54.318913 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:54.318920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:54.318922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:54.319064 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:54.319349 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:54.319352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:54.319354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:54.319709 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:54.319941 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:54.320153 (+ 212us) server_negotiation.cc:299] Negotiation successful
0809 04:52:54.320313 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:52:54.321280 20899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:52:54.422600 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 20686
I20260809 04:52:54.449455 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 20914
I20260809 04:52:54.462136 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:54.450261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.31.93.3:46861)
0809 04:52:54.450459 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:54.450465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:54.450503 (+ 38us) server_negotiation.cc:407] Connection header received
0809 04:52:54.450706 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:54.450712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:54.450784 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:54.450901 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:54.452136 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:54.452870 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:54.454142 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:54.454629 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:54.454742 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:54.457059 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:54.457073 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:54.457092 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:54.457124 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:54.459543 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:54.460188 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:54.460195 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:54.460198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:54.460333 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:54.460674 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:54.460679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:54.460682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:54.461030 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:54.461316 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:54.461628 (+ 312us) server_negotiation.cc:299] Negotiation successful
0809 04:52:54.461841 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 04:52:54.462855 21105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:52:54.547864 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 20914
I20260809 04:52:54.571138 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 20340
I20260809 04:52:54.572317 20431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:52:54.679455 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 20340
I20260809 04:52:54.712924 32116 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 04:52:55.056646 32116 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 04:52:55.056870 32116 mini_ranger.cc:66] Stopping Ranger...
I20260809 04:52:55.401191 32116 mini_ranger.cc:68] Stopped Ranger
2026-08-09 04:52:55.401 UTC [18909] LOG: received smart shutdown request
2026-08-09 04:52:55.404 UTC [18909] LOG: background worker "logical replication launcher" (PID 18927) exited with exit code 1
2026-08-09 04:52:55.404 UTC [18919] LOG: shutting down
2026-08-09 04:52:55.532 UTC [18909] LOG: database system is shut down
2026-08-09T04:52:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51200 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:52:55 dist-test-slave-4tdj krb5kdc[21206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:52:55 dist-test-slave-4tdj krb5kdc[21206](info): set up 2 sockets
Aug 09 04:52:55 dist-test-slave-4tdj krb5kdc[21206](info): commencing operation
krb5kdc: starting...
W20260809 04:52:56.836153 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:52:56 dist-test-slave-4tdj krb5kdc[21206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:52:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:52:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:52:57.059490 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:34327
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:42675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:34327,127.31.93.61:36493,127.31.93.60:37873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:52:57.376446 21225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:52:57.376708 21225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:52:57.376794 21225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:52:57.383955 21225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:52:57.384084 21225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:52:57.384160 21225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:52:57.384238 21225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:52:57.384299 21225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:52:57.394104 21225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:42675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:34327,127.31.93.61:36493,127.31.93.60:37873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:34327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:52:57.395387 21225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:52:57.396631 21225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:52:57.407481 21233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:57.407505 21237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:57.407676 21234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:52:57.409334 21225 server_base.cc:1034] running on GCE node
I20260809 04:52:57.410072 21225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:52:57.411463 21225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:52:57.412667 21225 hybrid_clock.cc:648] HybridClock initialized: now 1786251177412622 us; error 40 us; skew 500 ppm
I20260809 04:52:57.412978 21225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:52:57 dist-test-slave-4tdj krb5kdc[21206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251177, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:52:57.416074 21225 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:52:57.417532 21225 webserver.cc:460] Webserver started at http://127.31.93.62:43933/ using document root <none> and password file <none>
I20260809 04:52:57.418213 21225 fs_manager.cc:359] Metadata directory not provided
I20260809 04:52:57.418352 21225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:52:57.418643 21225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:52:57.421573 21225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d"
format_stamp: "Formatted at 2026-08-09 04:52:57 on dist-test-slave-4tdj"
server_key: "b501cd5614811675c9eb56e64fb010cf"
server_key_iv: "202e17887d50886f5795692978c26eeb"
server_key_version: "encryptionkey@0"
I20260809 04:52:57.422477 21225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d"
format_stamp: "Formatted at 2026-08-09 04:52:57 on dist-test-slave-4tdj"
server_key: "b501cd5614811675c9eb56e64fb010cf"
server_key_iv: "202e17887d50886f5795692978c26eeb"
server_key_version: "encryptionkey@0"
I20260809 04:52:57.427492 21225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 04:52:57.430773 21249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:57.432031 21225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 04:52:57.432204 21225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d"
format_stamp: "Formatted at 2026-08-09 04:52:57 on dist-test-slave-4tdj"
server_key: "b501cd5614811675c9eb56e64fb010cf"
server_key_iv: "202e17887d50886f5795692978c26eeb"
server_key_version: "encryptionkey@0"
I20260809 04:52:57.432384 21225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:52:57.456406 21225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:52:57.460248 21225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:52:57.460502 21225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:52:57.478506 21225 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:34327
I20260809 04:52:57.478559 21327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:34327 every 8 connection(s)
I20260809 04:52:57.479815 21225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:52:57.483585 21328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:52:57.487955 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 21225
I20260809 04:52:57.488160 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:52:57.488767 32116 external_mini_cluster.cc:1428] Setting key 9f2be77c3eab3c5fe3c17ccc659a3ae5
I20260809 04:52:57.490414 21328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:52:57 dist-test-slave-4tdj krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251177, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.93.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.61@KRBTEST.COM" created.
I20260809 04:52:57.512610 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:57.493407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:34327 (local address 127.0.0.1:51572)
0809 04:52:57.494285 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:57.494361 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:57.496459 (+ 2098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:57.497388 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:57.497393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:57.497956 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:57.498350 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:57.498372 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:57.499980 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:57.499987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:57.501082 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:57.501095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:57.502040 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:57.502048 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:57.502239 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:57.502902 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:57.502927 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:57.505102 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:57.508329 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:57.508341 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:57.508353 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:57.508766 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:57.509189 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:57.509194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:57.509199 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:57.509360 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:57.510136 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:57.510144 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:57.510414 (+ 270us) client_negotiation.cc:769] Sending connection context
0809 04:52:57.510681 (+ 267us) client_negotiation.cc:240] Negotiation successful
0809 04:52:57.511264 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":142,"thread_start_us":376,"threads_started":1}
I20260809 04:52:57.512631 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:57.494773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.31.93.62:34327)
0809 04:52:57.495465 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:57.495476 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:57.495526 (+ 50us) server_negotiation.cc:407] Connection header received
0809 04:52:57.496836 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:57.496843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:57.497141 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:57.497352 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:57.498561 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:57.499819 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:57.501224 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:57.501797 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:57.501988 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:57.505338 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:57.505346 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:57.505357 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:57.505394 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:57.508199 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:57.508901 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:57.508908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:57.508910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:57.509079 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:57.509478 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:57.509482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:57.509484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:57.509984 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:57.510189 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:57.510845 (+ 656us) server_negotiation.cc:299] Negotiation successful
0809 04:52:57.511133 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":352,"threads_started":1}
I20260809 04:52:57.515792 21328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:52:57.516878 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:57.516463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:36493 (local address 127.0.0.1:36596)
0809 04:52:57.516651 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:57.516763 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260809 04:52:57.517167 21251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.93.61:36493: connect: Connection refused (error 111)
I20260809 04:52:57.518795 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:57.518443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:36493 (local address 127.0.0.1:36600)
0809 04:52:57.518582 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:57.518651 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 04:52:57.518996 21328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.61:36493: Network error: Client connection negotiation failed: client connection to 127.31.93.61:36493: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.61.keytab.
Entry for principal kudu/127.31.93.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.61.keytab.
I20260809 04:52:57.539428 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.61:36493
--webserver_interface=127.31.93.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.61.keytab
--principal=kudu/127.31.93.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:42675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:34327,127.31.93.61:36493,127.31.93.60:37873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:52:57.568382 21328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } attempt: 1
I20260809 04:52:57.569480 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:57.569047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:36493 (local address 127.0.0.1:36602)
0809 04:52:57.569239 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:57.569348 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 04:52:57.570544 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:57.570166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:36493 (local address 127.0.0.1:36618)
0809 04:52:57.570282 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:57.570383 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 04:52:57.570739 21328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.61:36493: Network error: Client connection negotiation failed: client connection to 127.31.93.61:36493: connect: Connection refused (error 111)
I20260809 04:52:57.638983 21328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } attempt: 2
I20260809 04:52:57.640148 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:57.639717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:36493 (local address 127.0.0.1:36626)
0809 04:52:57.639908 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:57.640002 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 04:52:57.641142 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:57.640825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:36493 (local address 127.0.0.1:36628)
0809 04:52:57.640940 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:57.641020 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 04:52:57.641347 21328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.61:36493: Network error: Client connection negotiation failed: client connection to 127.31.93.61:36493: connect: Connection refused (error 111)
I20260809 04:52:57.732599 21328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } attempt: 3
I20260809 04:52:57.733837 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:57.733393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:36493 (local address 127.0.0.1:36636)
0809 04:52:57.733592 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:57.733703 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260809 04:52:57.734862 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:57.734543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:36493 (local address 127.0.0.1:36638)
0809 04:52:57.734662 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:57.734732 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 04:52:57.735065 21328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.61:36493: Network error: Client connection negotiation failed: client connection to 127.31.93.61:36493: connect: Connection refused (error 111)
W20260809 04:52:57.868752 21342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:52:57.869027 21342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:52:57.869165 21342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:52:57.876534 21342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:52:57.876646 21342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:52:57.876731 21342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:52:57.876809 21342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:52:57.876874 21342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:52:57.878307 21328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } attempt: 4
I20260809 04:52:57.879446 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:57.878917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:36493 (local address 127.0.0.1:33710)
0809 04:52:57.879229 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:57.879318 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260809 04:52:57.880533 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:57.880202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:36493 (local address 127.0.0.1:33722)
0809 04:52:57.880324 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:57.880417 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 04:52:57.880728 21328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.61:36493: Network error: Client connection negotiation failed: client connection to 127.31.93.61:36493: connect: Connection refused (error 111)
I20260809 04:52:57.887518 21342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:42675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:34327,127.31.93.61:36493,127.31.93.60:37873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.61.keytab
--principal=kudu/127.31.93.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.61:36493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.93.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:52:57.888792 21342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:52:57.890090 21342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:52:57.901403 21349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:57.901427 21350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:57.901594 21353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:52:57.902452 21342 server_base.cc:1034] running on GCE node
I20260809 04:52:57.903179 21342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:52:57.904451 21342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:52:57.905644 21342 hybrid_clock.cc:648] HybridClock initialized: now 1786251177905588 us; error 47 us; skew 500 ppm
I20260809 04:52:57.905951 21342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:52:57 dist-test-slave-4tdj krb5kdc[21206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251177, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:52:57.908843 21342 init.cc:377] Logged in from keytab as kudu/127.31.93.61@KRBTEST.COM (short username kudu)
I20260809 04:52:57.910147 21342 webserver.cc:460] Webserver started at http://127.31.93.61:36605/ using document root <none> and password file <none>
I20260809 04:52:57.910802 21342 fs_manager.cc:359] Metadata directory not provided
I20260809 04:52:57.910915 21342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:52:57.911196 21342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:52:57.913941 21342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/data/instance:
uuid: "fd8132acc8d1441180da256b9ce6930c"
format_stamp: "Formatted at 2026-08-09 04:52:57 on dist-test-slave-4tdj"
server_key: "f644d77006213a2b37b1b9a0c8a39adc"
server_key_iv: "97fe9029aff5cd0f6d8a48ebe3179911"
server_key_version: "encryptionkey@0"
I20260809 04:52:57.914701 21342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/wal/instance:
uuid: "fd8132acc8d1441180da256b9ce6930c"
format_stamp: "Formatted at 2026-08-09 04:52:57 on dist-test-slave-4tdj"
server_key: "f644d77006213a2b37b1b9a0c8a39adc"
server_key_iv: "97fe9029aff5cd0f6d8a48ebe3179911"
server_key_version: "encryptionkey@0"
I20260809 04:52:57.919330 21342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260809 04:52:57.922472 21368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:57.923655 21342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 04:52:57.923813 21342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/wal
uuid: "fd8132acc8d1441180da256b9ce6930c"
format_stamp: "Formatted at 2026-08-09 04:52:57 on dist-test-slave-4tdj"
server_key: "f644d77006213a2b37b1b9a0c8a39adc"
server_key_iv: "97fe9029aff5cd0f6d8a48ebe3179911"
server_key_version: "encryptionkey@0"
I20260809 04:52:57.923974 21342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:52:57.948233 21342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:52:57.952126 21342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:52:57.952376 21342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:52:57.968886 21342 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.61:36493
I20260809 04:52:57.968927 21436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.61:36493 every 8 connection(s)
I20260809 04:52:57.970090 21342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/data/info.pb
I20260809 04:52:57.971242 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 21342
I20260809 04:52:57.971427 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-1/wal/instance
I20260809 04:52:57.972003 32116 external_mini_cluster.cc:1428] Setting key dc6efd5a2c0b10011d9b938ae289b0f6
I20260809 04:52:57.974398 21437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:52:57.982679 21437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:52:57 dist-test-slave-4tdj krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251177, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.61@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.93.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.60@KRBTEST.COM" created.
I20260809 04:52:58.001928 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:57.985175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60244 (local address 127.31.93.62:34327)
0809 04:52:57.985352 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:57.985356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:57.986651 (+ 1295us) server_negotiation.cc:407] Connection header received
0809 04:52:57.988416 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:57.988422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:57.988500 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:57.988597 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:57.990009 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:57.990725 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:57.992214 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:57.992715 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:57.992828 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:57.996418 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:57.996423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:57.996426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:57.996462 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:57.999062 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:57.999737 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:57.999743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:57.999747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:57.999893 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.000257 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.000261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.000265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.000615 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:58.000750 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:58.001380 (+ 630us) server_negotiation.cc:299] Negotiation successful
0809 04:52:58.001614 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 04:52:58.002737 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:57.985693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:34327 (local address 127.0.0.1:60244)
0809 04:52:57.986500 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:57.986537 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:57.988152 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:57.988820 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:57.988828 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:57.989318 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:57.989830 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:57.989845 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:57.990883 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:57.990887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:57.992066 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:57.992088 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:57.992946 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:57.992954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:57.993171 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:57.993826 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:57.993852 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:57.996189 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:57.999199 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:57.999206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:57.999218 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:57.999620 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:57.999981 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:57.999984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:57.999986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:58.000163 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:58.000715 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:58.000720 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:58.001041 (+ 321us) client_negotiation.cc:769] Sending connection context
0809 04:52:58.001275 (+ 234us) client_negotiation.cc:240] Negotiation successful
0809 04:52:58.001636 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":381,"threads_started":1}
I20260809 04:52:58.004472 21437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:52:58 dist-test-slave-4tdj krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251177, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.61@KRBTEST.COM for kudu/127.31.93.61@KRBTEST.COM
I20260809 04:52:58.017535 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.005176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:36493 (local address 127.0.0.1:33724)
0809 04:52:58.005372 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.005390 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:58.005486 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:58.006243 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:58.006248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:58.006681 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:58.006840 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:58.006851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.008145 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.008150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:58.008957 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:58.008967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.009715 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.009721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:58.009907 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.010397 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:58.010414 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:58.012422 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:58.015316 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:58.015321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:58.015325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:58.015646 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:58.015983 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:58.015988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:58.015992 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:58.016118 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:58.016669 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:58.016674 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:58.016804 (+ 130us) client_negotiation.cc:769] Sending connection context
0809 04:52:58.016918 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 04:52:58.017165 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 04:52:58.017795 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.005304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.31.93.61:36493)
0809 04:52:58.005847 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:58.005852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:58.005878 (+ 26us) server_negotiation.cc:407] Connection header received
0809 04:52:58.005919 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:58.005925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:58.006099 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:58.006205 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:58.006994 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.008009 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.009090 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.009586 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.009700 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.012608 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:58.012618 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:58.012630 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:58.012662 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:58.015205 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.015757 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.015764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.015766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.015886 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.016211 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.016216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.016219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.016534 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:58.016728 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:58.017082 (+ 354us) server_negotiation.cc:299] Negotiation successful
0809 04:52:58.017281 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":261,"threads_started":1}
I20260809 04:52:58.020120 21437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:52:58.021297 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.020880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36546)
0809 04:52:58.021052 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.021166 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 04:52:58.021564 21373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
I20260809 04:52:58.023159 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.022866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36558)
0809 04:52:58.022998 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.023058 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 04:52:58.023353 21437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:37873: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.60.keytab.
Entry for principal kudu/127.31.93.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.60.keytab.
I20260809 04:52:58.027077 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.60:37873
--webserver_interface=127.31.93.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.60.keytab
--principal=kudu/127.31.93.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:42675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:34327,127.31.93.61:36493,127.31.93.60:37873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:52:58.072626 21437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } attempt: 1
I20260809 04:52:58.073725 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.073242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36566)
0809 04:52:58.073430 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.073543 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 04:52:58.074806 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.074443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36574)
0809 04:52:58.074615 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.074697 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260809 04:52:58.075033 21437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:37873: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
I20260809 04:52:58.143245 21437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } attempt: 2
I20260809 04:52:58.144423 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.143964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36584)
0809 04:52:58.144165 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.144262 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 04:52:58.145463 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.145142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36586)
0809 04:52:58.145278 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.145349 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 04:52:58.145699 21437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:37873: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
I20260809 04:52:58.179960 21328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } attempt: 5
Aug 09 04:52:58 dist-test-slave-4tdj krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251177, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for kudu/127.31.93.61@KRBTEST.COM
I20260809 04:52:58.194885 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.180748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:36493 (local address 127.0.0.1:33728)
0809 04:52:58.180975 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.181009 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:58.181130 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:58.181616 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:58.181620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:58.182009 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:58.182227 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:58.182237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.183725 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.183730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:58.184759 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:58.184776 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.185785 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.185790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:58.185932 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.186629 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:58.186652 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:58.188782 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:58.192537 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:58.192542 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:58.192544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:58.192915 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:58.193313 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:58.193317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:58.193319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:58.193453 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:58.194067 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:58.194071 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:58.194209 (+ 138us) client_negotiation.cc:769] Sending connection context
0809 04:52:58.194370 (+ 161us) client_negotiation.cc:240] Negotiation successful
0809 04:52:58.194604 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 04:52:58.195083 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.180925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.31.93.61:36493)
0809 04:52:58.181131 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:58.181139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:58.181166 (+ 27us) server_negotiation.cc:407] Connection header received
0809 04:52:58.181237 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:58.181243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:58.181349 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:58.181473 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:58.182436 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.183537 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.184924 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.185620 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.185752 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.188988 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:58.188994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:58.189000 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:58.189039 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:58.192353 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.193040 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.193046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.193050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.193203 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.193563 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.193567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.193571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.193959 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:58.194116 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:58.194497 (+ 381us) server_negotiation.cc:299] Negotiation successful
0809 04:52:58.194743 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:52:58.196158 21328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:52:58.197213 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.196794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36594)
0809 04:52:58.196968 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.197067 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 04:52:58.198235 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.197934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36604)
0809 04:52:58.198057 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.198138 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 04:52:58.198451 21328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:37873: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
I20260809 04:52:58.236907 21437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } attempt: 3
I20260809 04:52:58.238111 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.237612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36616)
0809 04:52:58.237832 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.237942 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 04:52:58.239289 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.238821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36620)
0809 04:52:58.239076 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.239163 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260809 04:52:58.239519 21437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:37873: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
I20260809 04:52:58.249662 21328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } attempt: 1
I20260809 04:52:58.250689 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.250235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36632)
0809 04:52:58.250433 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.250534 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260809 04:52:58.251616 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.251285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36638)
0809 04:52:58.251428 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.251507 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 04:52:58.251825 21328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:37873: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
I20260809 04:52:58.320056 21328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } attempt: 2
I20260809 04:52:58.321208 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.320706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36642)
0809 04:52:58.320920 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.321039 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 04:52:58.322189 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.321835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36652)
0809 04:52:58.321989 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.322063 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 04:52:58.322425 21328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:37873: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
W20260809 04:52:58.345912 21451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:52:58.346163 21451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:52:58.346311 21451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:52:58.353590 21451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:52:58.353693 21451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:52:58.353770 21451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:52:58.353844 21451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:52:58.353905 21451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:52:58.363154 21451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:42675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:34327,127.31.93.61:36493,127.31.93.60:37873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.60.keytab
--principal=kudu/127.31.93.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.60:37873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.93.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:52:58.364326 21451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:52:58.365593 21451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:52:58.376674 21458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:58.376644 21460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:52:58.376907 21462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:52:58.377753 21451 server_base.cc:1034] running on GCE node
I20260809 04:52:58.378471 21451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:52:58.379670 21451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:52:58.380859 21451 hybrid_clock.cc:648] HybridClock initialized: now 1786251178380817 us; error 41 us; skew 500 ppm
I20260809 04:52:58.381160 21451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:52:58.382721 21437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } attempt: 4
Aug 09 04:52:58 dist-test-slave-4tdj krb5kdc[21206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251178, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:52:58.383706 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.383260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36666)
0809 04:52:58.383484 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.383563 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260809 04:52:58.384336 21451 init.cc:377] Logged in from keytab as kudu/127.31.93.60@KRBTEST.COM (short username kudu)
I20260809 04:52:58.384758 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.384375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36682)
0809 04:52:58.384532 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.384621 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 04:52:58.385075 21437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:37873: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
I20260809 04:52:58.386014 21451 webserver.cc:460] Webserver started at http://127.31.93.60:43839/ using document root <none> and password file <none>
I20260809 04:52:58.386888 21451 fs_manager.cc:359] Metadata directory not provided
I20260809 04:52:58.387027 21451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:52:58.387377 21451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:52:58.390698 21451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/data/instance:
uuid: "70d34064e4594da9a43220fa5ef4e43b"
format_stamp: "Formatted at 2026-08-09 04:52:58 on dist-test-slave-4tdj"
server_key: "b7bbe06aa2b126750b6a39c8a2bd3567"
server_key_iv: "c0ced66805eb9a61c5712850017f0024"
server_key_version: "encryptionkey@0"
I20260809 04:52:58.391479 21451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/wal/instance:
uuid: "70d34064e4594da9a43220fa5ef4e43b"
format_stamp: "Formatted at 2026-08-09 04:52:58 on dist-test-slave-4tdj"
server_key: "b7bbe06aa2b126750b6a39c8a2bd3567"
server_key_iv: "c0ced66805eb9a61c5712850017f0024"
server_key_version: "encryptionkey@0"
I20260809 04:52:58.396386 21451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 04:52:58.399811 21477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:52:58.401075 21451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260809 04:52:58.401216 21451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/wal
uuid: "70d34064e4594da9a43220fa5ef4e43b"
format_stamp: "Formatted at 2026-08-09 04:52:58 on dist-test-slave-4tdj"
server_key: "b7bbe06aa2b126750b6a39c8a2bd3567"
server_key_iv: "c0ced66805eb9a61c5712850017f0024"
server_key_version: "encryptionkey@0"
I20260809 04:52:58.401376 21451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:52:58.414245 21451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:52:58.418359 21451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:52:58.418622 21451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:52:58.428627 21328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } attempt: 3
I20260809 04:52:58.429811 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.429287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36692)
0809 04:52:58.429537 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.429645 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 04:52:58.431012 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.430593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36708)
0809 04:52:58.430783 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.430872 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260809 04:52:58.431275 21328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:37873: Network error: Client connection negotiation failed: client connection to 127.31.93.60:37873: connect: Connection refused (error 111)
I20260809 04:52:58.436578 21451 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.60:37873
I20260809 04:52:58.436615 21552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.60:37873 every 8 connection(s)
I20260809 04:52:58.437934 21451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/data/info.pb
I20260809 04:52:58.441603 21553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:52:58.448129 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 21451
I20260809 04:52:58.448319 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786250977734564-32116-0/minicluster-data/master-2/wal/instance
I20260809 04:52:58.447762 21553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:52:58.448882 32116 external_mini_cluster.cc:1428] Setting key 9d91ca40889b0c5f214013e288971f4d
Aug 09 04:52:58 dist-test-slave-4tdj krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251178, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.60@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
Aug 09 04:52:58 dist-test-slave-4tdj krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.60@KRBTEST.COM
I20260809 04:52:58.467120 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.451618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.31.93.62:34327)
0809 04:52:58.451828 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:58.451835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:58.452415 (+ 580us) server_negotiation.cc:407] Connection header received
0809 04:52:58.453558 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:58.453562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:58.453629 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:58.453706 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:58.455071 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.455805 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.457501 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.458144 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.458273 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.461501 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:58.461507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:58.461512 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:58.461552 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:58.464242 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.465007 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.465012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.465016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.465143 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.465542 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.465548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.465551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.465813 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:58.465968 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:58.466600 (+ 632us) server_negotiation.cc:299] Negotiation successful
0809 04:52:58.466812 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 04:52:58.468168 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.451576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:34327 (local address 127.0.0.1:60248)
0809 04:52:58.452256 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.452305 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:58.453326 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:58.453799 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:58.453805 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:58.454530 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:58.454900 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:58.454917 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.455958 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.455963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:58.457364 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:58.457376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.458392 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.458400 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:58.458655 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.459338 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:58.459364 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:58.461311 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:58.464364 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:58.464375 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:58.464385 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:58.464898 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:58.465260 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:58.465265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:58.465269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:58.465431 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:58.465937 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:58.465943 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:58.466223 (+ 280us) client_negotiation.cc:769] Sending connection context
0809 04:52:58.466520 (+ 297us) client_negotiation.cc:240] Negotiation successful
0809 04:52:58.466882 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":272,"threads_started":1}
I20260809 04:52:58.468559 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.450671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36710 (local address 127.31.93.60:37873)
0809 04:52:58.451317 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:58.451327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:58.451366 (+ 39us) server_negotiation.cc:407] Connection header received
0809 04:52:58.453316 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:58.453343 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:58.453629 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:58.453805 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:58.455253 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.456704 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.458262 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.459014 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.459101 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.463060 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:58.463067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:58.463081 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:58.463123 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:58.465699 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.466419 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.466426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.466430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.466583 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.466983 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.466988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.466991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.467425 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:58.467699 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:58.467969 (+ 270us) server_negotiation.cc:299] Negotiation successful
0809 04:52:58.468200 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":276,"threads_started":1}
I20260809 04:52:58.469972 21553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:52:58 dist-test-slave-4tdj krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251178, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.60@KRBTEST.COM for kudu/127.31.93.61@KRBTEST.COM
Aug 09 04:52:58 dist-test-slave-4tdj krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:52:58.484113 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.470675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:36493 (local address 127.0.0.1:33732)
0809 04:52:58.470974 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.470993 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:58.471097 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:58.471554 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:58.471557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:58.471856 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:58.471973 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:58.471980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.473262 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.473267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:58.474523 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:58.474535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.475432 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.475435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:58.475577 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.476025 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:58.476046 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:58.478234 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:58.481671 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:58.481677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:58.481681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:58.482161 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:58.482648 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:58.482653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:58.482657 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:58.482836 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:58.483398 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:58.483404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:58.483540 (+ 136us) client_negotiation.cc:769] Sending connection context
0809 04:52:58.483676 (+ 136us) client_negotiation.cc:240] Negotiation successful
0809 04:52:58.483863 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20260809 04:52:58.484370 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.470689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.31.93.61:36493)
0809 04:52:58.470862 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:58.470868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:58.471074 (+ 206us) server_negotiation.cc:407] Connection header received
0809 04:52:58.471248 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:58.471254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:58.471331 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:58.471400 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:58.472121 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.473135 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.474656 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.475317 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.475444 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.478451 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:58.478459 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:58.478465 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:58.478512 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:58.481533 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.482334 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.482340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.482343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.482512 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.482950 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.482955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.482959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.483273 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:58.483430 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:58.483788 (+ 358us) server_negotiation.cc:299] Negotiation successful
0809 04:52:58.484000 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 04:52:58.485435 21553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:52:58.488657 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.471284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60260 (local address 127.31.93.62:34327)
0809 04:52:58.471452 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:58.471458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:58.471473 (+ 15us) server_negotiation.cc:407] Connection header received
0809 04:52:58.471514 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:58.471517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:58.471566 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:58.471639 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:58.473030 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.474055 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.475787 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.476352 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.476480 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.481511 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:58.481518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:58.481523 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:58.481566 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:58.485903 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.486699 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.486705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.486708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.486903 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.487352 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.487358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.487362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.487627 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:58.487916 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:58.488070 (+ 154us) server_negotiation.cc:299] Negotiation successful
0809 04:52:58.488292 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Aug 09 04:52:58 dist-test-slave-4tdj krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251178, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.60@KRBTEST.COM for kudu/127.31.93.60@KRBTEST.COM
Aug 09 04:52:58 dist-test-slave-4tdj krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.61@KRBTEST.COM
I20260809 04:52:58.498682 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.486254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36720)
0809 04:52:58.486417 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.486436 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:58.486544 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:58.487009 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:58.487014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:58.487448 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:58.487580 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:58.487590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.488908 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.488912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:58.489936 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:58.489949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.490769 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.490776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:58.490961 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.491465 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:58.491482 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:58.493453 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:58.496463 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:58.496477 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:58.496482 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:58.496853 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:58.497213 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:58.497218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:58.497223 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:58.497350 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:58.497903 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:58.497907 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:58.498020 (+ 113us) client_negotiation.cc:769] Sending connection context
0809 04:52:58.498152 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 04:52:58.498394 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 04:52:58.499145 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.486576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36720 (local address 127.31.93.60:37873)
0809 04:52:58.486723 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:58.486729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:58.486745 (+ 16us) server_negotiation.cc:407] Connection header received
0809 04:52:58.486790 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:58.486795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:58.486860 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:58.486966 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:58.487721 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.488685 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.490079 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.490625 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.490740 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.493632 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:58.493637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:58.493640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:58.493668 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:58.496339 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.496959 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.496965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.496977 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.497108 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.497452 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.497458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.497462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.497778 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:58.497932 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:58.498281 (+ 349us) server_negotiation.cc:299] Negotiation successful
0809 04:52:58.498567 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":55}
I20260809 04:52:58.504216 21553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b: Bootstrap starting.
I20260809 04:52:58.504733 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.490550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33740 (local address 127.31.93.61:36493)
0809 04:52:58.490689 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:58.490693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:58.490706 (+ 13us) server_negotiation.cc:407] Connection header received
0809 04:52:58.490741 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:58.490744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:58.490791 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:58.490868 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:58.492131 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.493049 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.494233 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.494922 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.495067 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.499433 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:58.499440 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:58.499445 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:58.499499 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:58.502074 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.502853 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.502857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.502859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.503003 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.503417 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.503421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.503423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.503694 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:58.503915 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:58.504195 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 04:52:58.504416 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 04:52:58.507591 21553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b: Neither blocks nor log segments found. Creating new log.
I20260809 04:52:58.508563 21553 log.cc:824] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b: Log is configured to *not* fsync() on all Append() calls
I20260809 04:52:58.509419 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:52:58.511571 21553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b: No bootstrap required, opened a new log
I20260809 04:52:58.522449 21553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } }
I20260809 04:52:58.522871 21553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:52:58.522986 21553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d34064e4594da9a43220fa5ef4e43b, State: Initialized, Role: FOLLOWER
I20260809 04:52:58.523741 21553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } }
I20260809 04:52:58.529731 21570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } } }
I20260809 04:52:58.530997 21553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:52:58.532351 21570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:52:58.542443 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.515588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60276 (local address 127.31.93.62:34327)
0809 04:52:58.515985 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:58.515993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:58.521969 (+ 5976us) server_negotiation.cc:407] Connection header received
0809 04:52:58.522521 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:58.522528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:58.522623 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:58.522704 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:58.525906 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.527005 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.530504 (+ 3499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.531170 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.531273 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.535182 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:58.535190 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:58.535196 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:58.535264 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:58.538684 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.539474 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.539480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.539484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.539651 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.540096 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.540102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.540105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.540455 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:58.540708 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:58.541684 (+ 976us) server_negotiation.cc:299] Negotiation successful
0809 04:52:58.541962 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
I20260809 04:52:58.543135 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.514783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.31.93.61:36493)
0809 04:52:58.514946 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:58.514952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:58.526516 (+ 11564us) server_negotiation.cc:407] Connection header received
0809 04:52:58.526965 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:58.526983 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:58.527065 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:58.527133 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:58.529744 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.530826 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.532810 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.533409 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.533535 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.537169 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:58.537176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:58.537180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:58.537217 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:58.540262 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.541118 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.541123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.541126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.541276 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.541735 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.541740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.541744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.541996 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:58.542235 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:58.542573 (+ 338us) server_negotiation.cc:299] Negotiation successful
0809 04:52:58.542785 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:52:58.545413 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.515360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36724 (local address 127.31.93.60:37873)
0809 04:52:58.515817 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:58.515824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:58.524757 (+ 8933us) server_negotiation.cc:407] Connection header received
0809 04:52:58.524903 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:58.524909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:58.524985 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:58.525072 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:58.527671 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.528807 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.530933 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.531427 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.531496 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.535947 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:58.535953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:58.535958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:58.535995 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:58.541002 (+ 5007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.543030 (+ 2028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.543035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.543038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.543303 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.543782 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.543788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.543790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.544169 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:58.544409 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:58.544947 (+ 538us) server_negotiation.cc:299] Negotiation successful
0809 04:52:58.545156 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302}
W20260809 04:52:58.553056 21584 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:52:58.553191 21584 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:52:58.549690 21451 master_runner.cc:186] Error getting master registration for 127.31.93.62:34327: OK, instance_id { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" instance_seqno: 1786251177473323 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 04:52:58.561228 21451 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 04:52:58.569787 21570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:52:58.570063 21570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } }
I20260809 04:52:58.571802 21570 leader_election.cc:288] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eefcaf907ccb4bc6896c4a6bf1fad36d (127.31.93.62:34327), fd8132acc8d1441180da256b9ce6930c (127.31.93.61:36493)
I20260809 04:52:58.608537 21328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } attempt: 4
Aug 09 04:52:58 dist-test-slave-4tdj krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251177, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for kudu/127.31.93.60@KRBTEST.COM
I20260809 04:52:58.625573 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.609183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.31.93.60:37873)
0809 04:52:58.609418 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:58.609424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:58.609452 (+ 28us) server_negotiation.cc:407] Connection header received
0809 04:52:58.609520 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:58.609525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:58.609600 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:58.609717 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:58.610604 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.611809 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.613583 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.614348 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.614500 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.618399 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:58.618406 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:58.618412 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:58.618454 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:58.622825 (+ 4371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.623618 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.623624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.623628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.623807 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.624165 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.624170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.624174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.624507 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:58.624652 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:58.625005 (+ 353us) server_negotiation.cc:299] Negotiation successful
0809 04:52:58.625217 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260809 04:52:58.625592 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.609128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36726)
0809 04:52:58.609303 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.609322 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:58.609426 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:58.609739 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:58.609744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:58.610233 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:58.610458 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:58.610471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.611943 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.611948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:58.613420 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:58.613435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.614517 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.614523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:58.614714 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.615359 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:58.615382 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:58.618178 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:58.622986 (+ 4808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:58.622992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:58.622996 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:58.623507 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:58.623916 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:58.623921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:58.623925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:58.624060 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:58.624645 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:58.624650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:58.624786 (+ 136us) client_negotiation.cc:769] Sending connection context
0809 04:52:58.624901 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 04:52:58.625186 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260809 04:52:58.631634 21328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d: Bootstrap starting.
I20260809 04:52:58.636767 21328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d: Neither blocks nor log segments found. Creating new log.
I20260809 04:52:58.638120 21328 log.cc:824] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d: Log is configured to *not* fsync() on all Append() calls
I20260809 04:52:58.641449 21328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d: No bootstrap required, opened a new log
I20260809 04:52:58.653836 21328 raft_consensus.cc:348] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } }
I20260809 04:52:58.654229 21328 raft_consensus.cc:374] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:52:58.654404 21328 raft_consensus.cc:729] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eefcaf907ccb4bc6896c4a6bf1fad36d, State: Initialized, Role: FOLLOWER
I20260809 04:52:58.655059 21328 consensus_queue.cc:260] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } }
I20260809 04:52:58.656050 21590 sys_catalog.cc:455] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } } }
I20260809 04:52:58.656447 21590 sys_catalog.cc:458] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:52:58.657387 21328 sys_catalog.cc:564] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:52:58.657414 21290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70d34064e4594da9a43220fa5ef4e43b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" is_pre_election: true
I20260809 04:52:58.657984 21290 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70d34064e4594da9a43220fa5ef4e43b in term 0.
I20260809 04:52:58.658602 21481 leader_election.cc:302] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70d34064e4594da9a43220fa5ef4e43b, eefcaf907ccb4bc6896c4a6bf1fad36d; no voters:
I20260809 04:52:58.659058 21570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:52:58.659219 21570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:52:58.659329 21570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:52:58.661545 21570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } }
I20260809 04:52:58.662549 21290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70d34064e4594da9a43220fa5ef4e43b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d"
I20260809 04:52:58.662755 21290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:52:58.662791 21570 leader_election.cc:288] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [CANDIDATE]: Term 1 election: Requested vote from peers eefcaf907ccb4bc6896c4a6bf1fad36d (127.31.93.62:34327), fd8132acc8d1441180da256b9ce6930c (127.31.93.61:36493)
I20260809 04:52:58.665899 21290 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70d34064e4594da9a43220fa5ef4e43b in term 1.
I20260809 04:52:58.668655 21481 leader_election.cc:302] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70d34064e4594da9a43220fa5ef4e43b, eefcaf907ccb4bc6896c4a6bf1fad36d; no voters:
I20260809 04:52:58.668944 21570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [term 1 FOLLOWER]: Leader election won for term 1
W20260809 04:52:58.670498 21488 tablet.cc:2369] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:52:58.681805 21570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [term 1 LEADER]: Becoming Leader. State: Replica: 70d34064e4594da9a43220fa5ef4e43b, State: Running, Role: LEADER
I20260809 04:52:58.682448 21570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } }
I20260809 04:52:58.684494 21437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } attempt: 5
I20260809 04:52:58.692955 21608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70d34064e4594da9a43220fa5ef4e43b. Latest consensus state: current_term: 1 leader_uuid: "70d34064e4594da9a43220fa5ef4e43b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } } }
I20260809 04:52:58.693387 21608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [sys.catalog]: This master's current role is: LEADER
Aug 09 04:52:58 dist-test-slave-4tdj krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251177, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.61@KRBTEST.COM for kudu/127.31.93.60@KRBTEST.COM
I20260809 04:52:58.680569 21225 master_runner.cc:186] Error getting master registration for 127.31.93.61:36493: OK, instance_id { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" instance_seqno: 1786251177964194 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 04:52:58.698341 21225 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 04:52:58.702503 21615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260809 04:52:58.702775 21611 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:52:58.702947 21611 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:52:58.704958 21615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:52:58.709129 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.685040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:37873 (local address 127.0.0.1:36736)
0809 04:52:58.685378 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0809 04:52:58.685398 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:52:58.685505 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:52:58.687225 (+ 1720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:52:58.687231 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:52:58.687814 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:52:58.688033 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:58.688045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.689580 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.689586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:58.690942 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:52:58.690957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.692113 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.692119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:52:58.692349 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.693022 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:52:58.693045 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:52:58.696161 (+ 3116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:52:58.702132 (+ 5971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:58.702138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:58.702143 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:58.703142 (+ 999us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:58.704033 (+ 891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:52:58.704067 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:52:58.704072 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:52:58.704371 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:52:58.705873 (+ 1502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:52:58.705909 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:52:58.706194 (+ 285us) client_negotiation.cc:769] Sending connection context
0809 04:52:58.706484 (+ 290us) client_negotiation.cc:240] Negotiation successful
0809 04:52:58.707194 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215}
I20260809 04:52:58.710480 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:52:58.685693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.31.93.60:37873)
0809 04:52:58.686449 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0809 04:52:58.686454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:52:58.686472 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:52:58.686520 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:52:58.686525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:52:58.686593 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:52:58.686680 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:52:58.688227 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.689391 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.691119 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:52:58.691765 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:52:58.691882 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:52:58.696381 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:52:58.696389 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:52:58.696394 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:52:58.696455 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:52:58.701958 (+ 5503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.703425 (+ 1467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.703432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.703436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.703785 (+ 349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:52:58.704739 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:52:58.704785 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:52:58.704790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:52:58.705625 (+ 835us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:52:58.707273 (+ 1648us) server_negotiation.cc:1035] Waiting for connection context
0809 04:52:58.708094 (+ 821us) server_negotiation.cc:299] Negotiation successful
0809 04:52:58.709225 (+ 1131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645}
W20260809 04:52:58.713218 21260 tablet.cc:2369] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:52:58.716250 21437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c: Bootstrap starting.
I20260809 04:52:58.720245 21437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c: Neither blocks nor log segments found. Creating new log.
I20260809 04:52:58.721514 21437 log.cc:824] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c: Log is configured to *not* fsync() on all Append() calls
I20260809 04:52:58.725382 21437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c: No bootstrap required, opened a new log
I20260809 04:52:58.746423 21437 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } }
I20260809 04:52:58.746869 21437 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:52:58.746984 21437 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8132acc8d1441180da256b9ce6930c, State: Initialized, Role: FOLLOWER
I20260809 04:52:58.747186 21290 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d [term 1 FOLLOWER]: Refusing update from remote peer 70d34064e4594da9a43220fa5ef4e43b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:52:58.747625 21437 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } }
I20260809 04:52:58.748390 21608 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:52:58.758891 21618 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } } }
I20260809 04:52:58.759296 21618 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:52:58.760213 21437 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:52:58.760574 21403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:52:58.762594 21405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70d34064e4594da9a43220fa5ef4e43b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8132acc8d1441180da256b9ce6930c" is_pre_election: true
I20260809 04:52:58.764216 21404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70d34064e4594da9a43220fa5ef4e43b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8132acc8d1441180da256b9ce6930c"
I20260809 04:52:58.767158 21403 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c [term 1 FOLLOWER]: Refusing update from remote peer 70d34064e4594da9a43220fa5ef4e43b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:52:58.767997 21608 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:52:58.793006 21590 sys_catalog.cc:455] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70d34064e4594da9a43220fa5ef4e43b. Latest consensus state: current_term: 1 leader_uuid: "70d34064e4594da9a43220fa5ef4e43b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } } }
I20260809 04:52:58.793375 21590 sys_catalog.cc:458] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:52:58.796672 21616 mvcc.cc:205] Tried to move back new op lower bound from 7316484828114649088 to 7316484827896938496. Current Snapshot: MvccSnapshot[applied={T|T < 7316484828114649088}]
I20260809 04:52:58.805251 21608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70d34064e4594da9a43220fa5ef4e43b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } } }
I20260809 04:52:58.805495 21608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [sys.catalog]: This master's current role is: LEADER
I20260809 04:52:58.807525 21615 catalog_manager.cc:1347] Generated new cluster ID: 0041ac1a1a7d4b0a943944f24d311ac5
I20260809 04:52:58.807631 21615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:52:58.807755 21590 sys_catalog.cc:455] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70d34064e4594da9a43220fa5ef4e43b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } } }
I20260809 04:52:58.808012 21590 sys_catalog.cc:458] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:52:58.815606 21618 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70d34064e4594da9a43220fa5ef4e43b. Latest consensus state: current_term: 1 leader_uuid: "70d34064e4594da9a43220fa5ef4e43b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } } }
I20260809 04:52:58.815896 21618 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:52:58.818881 21570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70d34064e4594da9a43220fa5ef4e43b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } } }
I20260809 04:52:58.819208 21570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b [sys.catalog]: This master's current role is: LEADER
I20260809 04:52:58.822016 21623 mvcc.cc:205] Tried to move back new op lower bound from 7316484828114649088 to 7316484827896938496. Current Snapshot: MvccSnapshot[applied={T|T < 7316484828114649088}]
I20260809 04:52:58.826784 21618 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70d34064e4594da9a43220fa5ef4e43b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eefcaf907ccb4bc6896c4a6bf1fad36d" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34327 } } peers { permanent_uuid: "fd8132acc8d1441180da256b9ce6930c" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 36493 } } peers { permanent_uuid: "70d34064e4594da9a43220fa5ef4e43b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 37873 } } }
I20260809 04:52:58.826996 21618 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c [sys.catalog]: This master's current role is: FOLLOWER
W20260809 04:52:58.847456 21642 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:52:58.847632 21642 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:52:58.876545 21615 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:52:58.878449 21615 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:52:58.905720 21615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b: Generated new TSK 0
I20260809 04:52:58.906951 21615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:52:59.705102 21611 catalog_manager.cc:1259] Loaded cluster ID: 0041ac1a1a7d4b0a943944f24d311ac5
I20260809 04:52:59.705209 21611 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d: loading cluster ID for follower catalog manager: success
I20260809 04:52:59.707885 21611 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d: acquiring CA information for follower catalog manager: success
I20260809 04:52:59.709641 21611 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eefcaf907ccb4bc6896c4a6bf1fad36d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 04:52:59.849390 21642 catalog_manager.cc:1259] Loaded cluster ID: 0041ac1a1a7d4b0a943944f24d311ac5
I20260809 04:52:59.849490 21642 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c: loading cluster ID for follower catalog manager: success
I20260809 04:52:59.851927 21642 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c: acquiring CA information for follower catalog manager: success
I20260809 04:52:59.853654 21642 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd8132acc8d1441180da256b9ce6930c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 04:53:00.091434 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 21225
I20260809 04:53:00.093513 21319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:00.202672 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 21225
I20260809 04:53:00.229871 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 21342
I20260809 04:53:00.231024 21431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:00.338630 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 21342
I20260809 04:53:00.358824 21653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:00.358076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:34327 (local address 127.0.0.1:60282)
0809 04:53:00.358518 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:00.358645 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":232,"threads_started":1}
W20260809 04:53:00.359083 21481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.93.62:34327: connect: Connection refused (error 111)
I20260809 04:53:00.363399 21653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:00.363035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:34327 (local address 127.0.0.1:60298)
0809 04:53:00.363194 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:00.363276 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 04:53:00.363588 21481 consensus_peers.cc:487] T 00000000000000000000000000000000 P 70d34064e4594da9a43220fa5ef4e43b -> Peer eefcaf907ccb4bc6896c4a6bf1fad36d (127.31.93.62:34327): Couldn't send request to peer eefcaf907ccb4bc6896c4a6bf1fad36d. Status: Network error: Client connection negotiation failed: client connection to 127.31.93.62:34327: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 04:53:00.365434 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 21451
I20260809 04:53:00.366372 21548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:00.473030 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 21451
2026-08-09T04:53:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4845 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:53:00 dist-test-slave-4tdj krb5kdc[21662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:53:00 dist-test-slave-4tdj krb5kdc[21662](info): set up 2 sockets
Aug 09 04:53:00 dist-test-slave-4tdj krb5kdc[21662](info): commencing operation
krb5kdc: starting...
W20260809 04:53:01.676717 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:53:01 dist-test-slave-4tdj krb5kdc[21662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:53:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:53:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:53:01.913005 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:34447
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:34447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:53:02.232218 21679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:02.232502 21679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:02.232646 21679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:02.239638 21679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:53:02.239748 21679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:02.239826 21679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:02.239905 21679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:53:02.239967 21679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:53:02.249380 21679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:34447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:34447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:02.250679 21679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:02.251930 21679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:02.262984 21687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:02.262938 21689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:02.263018 21692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:02.263748 21679 server_base.cc:1034] running on GCE node
I20260809 04:53:02.264431 21679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:02.265799 21679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:02.266992 21679 hybrid_clock.cc:648] HybridClock initialized: now 1786251182266937 us; error 48 us; skew 500 ppm
I20260809 04:53:02.267318 21679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:02 dist-test-slave-4tdj krb5kdc[21662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251182, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:02.272962 21679 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:53:02.274370 21679 webserver.cc:460] Webserver started at http://127.31.93.62:35929/ using document root <none> and password file <none>
I20260809 04:53:02.275012 21679 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:02.275110 21679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:02.275355 21679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:53:02.278183 21679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "9c87a0a2b48d421fb4e7713d7a08f94b"
format_stamp: "Formatted at 2026-08-09 04:53:02 on dist-test-slave-4tdj"
server_key: "157be4052725f3152b5eb75e9de1cd88"
server_key_iv: "f99cb0f7145bc8d06bfc40b99580aa20"
server_key_version: "encryptionkey@0"
I20260809 04:53:02.278910 21679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "9c87a0a2b48d421fb4e7713d7a08f94b"
format_stamp: "Formatted at 2026-08-09 04:53:02 on dist-test-slave-4tdj"
server_key: "157be4052725f3152b5eb75e9de1cd88"
server_key_iv: "f99cb0f7145bc8d06bfc40b99580aa20"
server_key_version: "encryptionkey@0"
I20260809 04:53:02.283767 21679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 04:53:02.286966 21704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:02.288223 21679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:53:02.288364 21679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "9c87a0a2b48d421fb4e7713d7a08f94b"
format_stamp: "Formatted at 2026-08-09 04:53:02 on dist-test-slave-4tdj"
server_key: "157be4052725f3152b5eb75e9de1cd88"
server_key_iv: "f99cb0f7145bc8d06bfc40b99580aa20"
server_key_version: "encryptionkey@0"
I20260809 04:53:02.288519 21679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:02.303540 21679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:02.308173 21679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:02.308439 21679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:02.324471 21679 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:34447
I20260809 04:53:02.324501 21776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:34447 every 8 connection(s)
I20260809 04:53:02.325784 21679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:53:02.326282 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 21679
I20260809 04:53:02.326519 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:53:02.327338 32116 external_mini_cluster.cc:1428] Setting key 3f51ce2f0d0fd93f01749d74b7cbe7a2
I20260809 04:53:02.329973 21780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:53:02.340080 21780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b: Bootstrap starting.
Aug 09 04:53:02 dist-test-slave-4tdj krb5kdc[21662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:53:02.343456 21780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b: Neither blocks nor log segments found. Creating new log.
I20260809 04:53:02.344489 21780 log.cc:824] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:02.347606 21780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b: No bootstrap required, opened a new log
I20260809 04:53:02.348412 21787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:02.329097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56384 (local address 127.31.93.62:34447)
0809 04:53:02.329838 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:02.329850 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:02.329896 (+ 46us) server_negotiation.cc:407] Connection header received
0809 04:53:02.331586 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:02.331611 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:02.331964 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:02.332327 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:02.333824 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:02.335193 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:02.336500 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:02.337170 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:02.337357 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:02.341522 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:02.341529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:02.341545 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:02.341585 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:02.344458 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:02.345282 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:02.345291 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:02.345302 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:02.345515 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:02.345878 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:02.345882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:02.345884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:02.346400 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:02.346620 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:02.347056 (+ 436us) server_negotiation.cc:299] Negotiation successful
0809 04:53:02.347365 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":342,"threads_started":1}
I20260809 04:53:02.351926 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:53:02.356117 21780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c87a0a2b48d421fb4e7713d7a08f94b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34447 } }
I20260809 04:53:02.356426 21780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:53:02.356506 21780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c87a0a2b48d421fb4e7713d7a08f94b, State: Initialized, Role: FOLLOWER
I20260809 04:53:02.356961 21780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c87a0a2b48d421fb4e7713d7a08f94b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34447 } }
I20260809 04:53:02.357147 21780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:53:02.357241 21780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:53:02.357355 21780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:53:02.359313 21780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c87a0a2b48d421fb4e7713d7a08f94b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34447 } }
I20260809 04:53:02.359724 21780 leader_election.cc:302] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c87a0a2b48d421fb4e7713d7a08f94b; no voters:
I20260809 04:53:02.360198 21780 leader_election.cc:288] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:53:02.360311 21790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:53:02.360956 21790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [term 1 LEADER]: Becoming Leader. State: Replica: 9c87a0a2b48d421fb4e7713d7a08f94b, State: Running, Role: LEADER
I20260809 04:53:02.361445 21790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c87a0a2b48d421fb4e7713d7a08f94b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34447 } }
I20260809 04:53:02.361953 21780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:53:02.363925 21791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c87a0a2b48d421fb4e7713d7a08f94b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c87a0a2b48d421fb4e7713d7a08f94b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34447 } } }
I20260809 04:53:02.363898 21792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c87a0a2b48d421fb4e7713d7a08f94b. Latest consensus state: current_term: 1 leader_uuid: "9c87a0a2b48d421fb4e7713d7a08f94b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c87a0a2b48d421fb4e7713d7a08f94b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34447 } } }
I20260809 04:53:02.364358 21792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [sys.catalog]: This master's current role is: LEADER
I20260809 04:53:02.366178 21791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b [sys.catalog]: This master's current role is: LEADER
I20260809 04:53:02.366820 21795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:53:02.372087 21795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:53:02.380858 21795 catalog_manager.cc:1347] Generated new cluster ID: bb3e3998da5f4e73b209359d99bf3340
I20260809 04:53:02.380959 21795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:53:02.422549 21795 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:53:02.423852 21795 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:53:02.429553 21795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c87a0a2b48d421fb4e7713d7a08f94b: Generated new TSK 0
I20260809 04:53:02.430312 21795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:53:02.464604 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 21679
I20260809 04:53:02.465844 21767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:02.559163 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 21679
2026-08-09T04:53:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2082 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:53:02 dist-test-slave-4tdj krb5kdc[21824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:53:02 dist-test-slave-4tdj krb5kdc[21824](info): set up 2 sockets
Aug 09 04:53:02 dist-test-slave-4tdj krb5kdc[21824](info): commencing operation
krb5kdc: starting...
W20260809 04:53:03.762956 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:53:03 dist-test-slave-4tdj krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:53:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:53:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:53:03.984946 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:44725
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:44725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:53:04.300423 21841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:04.300700 21841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:04.300800 21841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:04.307806 21841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:53:04.307904 21841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:04.307955 21841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:04.308010 21841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:53:04.308054 21841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:53:04.317482 21841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:44725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:44725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:04.318784 21841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:04.320011 21841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:04.331424 21846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:04.331350 21847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:04.331434 21850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:04.332283 21841 server_base.cc:1034] running on GCE node
I20260809 04:53:04.333006 21841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:04.334347 21841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:04.335552 21841 hybrid_clock.cc:648] HybridClock initialized: now 1786251184335503 us; error 37 us; skew 500 ppm
I20260809 04:53:04.335870 21841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:04 dist-test-slave-4tdj krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251184, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:04.338932 21841 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:53:04.340339 21841 webserver.cc:460] Webserver started at http://127.31.93.62:33647/ using document root <none> and password file <none>
I20260809 04:53:04.341005 21841 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:04.341143 21841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:04.341446 21841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:53:04.344326 21841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c"
format_stamp: "Formatted at 2026-08-09 04:53:04 on dist-test-slave-4tdj"
server_key: "57ea955e214b39468e3982c4fdca4242"
server_key_iv: "2606f6902ae7924d4097eca2d8d50fc2"
server_key_version: "encryptionkey@0"
I20260809 04:53:04.345064 21841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c"
format_stamp: "Formatted at 2026-08-09 04:53:04 on dist-test-slave-4tdj"
server_key: "57ea955e214b39468e3982c4fdca4242"
server_key_iv: "2606f6902ae7924d4097eca2d8d50fc2"
server_key_version: "encryptionkey@0"
I20260809 04:53:04.349943 21841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 04:53:04.353329 21859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:04.354626 21841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 04:53:04.354789 21841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c"
format_stamp: "Formatted at 2026-08-09 04:53:04 on dist-test-slave-4tdj"
server_key: "57ea955e214b39468e3982c4fdca4242"
server_key_iv: "2606f6902ae7924d4097eca2d8d50fc2"
server_key_version: "encryptionkey@0"
I20260809 04:53:04.354966 21841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:04.365105 21841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:04.368995 21841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:04.369249 21841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:04.387969 21841 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:44725
I20260809 04:53:04.387988 21930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:44725 every 8 connection(s)
I20260809 04:53:04.389475 21841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:53:04.392046 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 21841
I20260809 04:53:04.392252 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:53:04.392918 32116 external_mini_cluster.cc:1428] Setting key 7dc0bf740b61136ca413a8eed7e06868
I20260809 04:53:04.393666 21931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 04:53:04 dist-test-slave-4tdj krb5kdc[21824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:53:04.409996 21931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c: Bootstrap starting.
I20260809 04:53:04.413920 21931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c: Neither blocks nor log segments found. Creating new log.
I20260809 04:53:04.415097 21931 log.cc:824] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:04.419112 21936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:04.394914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44254 (local address 127.31.93.62:44725)
0809 04:53:04.395693 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:04.395704 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:04.395744 (+ 40us) server_negotiation.cc:407] Connection header received
0809 04:53:04.397365 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:04.397390 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:04.397739 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:04.398024 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:04.400113 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:04.402042 (+ 1929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:04.403452 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:04.404457 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:04.404630 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:04.410420 (+ 5790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:04.410432 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:04.410449 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:04.410495 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:04.414347 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:04.415277 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:04.415286 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:04.415294 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:04.415488 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:04.415921 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:04.415926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:04.415930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:04.416631 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:04.416949 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:04.417420 (+ 471us) server_negotiation.cc:299] Negotiation successful
0809 04:53:04.417855 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":326,"threads_started":1}
I20260809 04:53:04.421212 21931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c: No bootstrap required, opened a new log
I20260809 04:53:04.428354 21931 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } }
I20260809 04:53:04.428748 21931 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:53:04.428836 21931 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd94ed2317924c2c9c1dfd6caf15aa3c, State: Initialized, Role: FOLLOWER
I20260809 04:53:04.429387 21931 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } }
I20260809 04:53:04.429620 21931 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:53:04.429744 21931 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:53:04.429896 21931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:53:04.431969 21931 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } }
I20260809 04:53:04.432514 21931 leader_election.cc:302] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd94ed2317924c2c9c1dfd6caf15aa3c; no voters:
I20260809 04:53:04.433049 21931 leader_election.cc:288] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:53:04.433183 21943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:53:04.433789 21943 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 1 LEADER]: Becoming Leader. State: Replica: fd94ed2317924c2c9c1dfd6caf15aa3c, State: Running, Role: LEADER
I20260809 04:53:04.434319 21943 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } }
I20260809 04:53:04.434934 21931 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:53:04.437781 21944 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } } }
I20260809 04:53:04.438150 21944 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [sys.catalog]: This master's current role is: LEADER
I20260809 04:53:04.441100 21961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:53:04.440979 21946 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd94ed2317924c2c9c1dfd6caf15aa3c. Latest consensus state: current_term: 1 leader_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } } }
I20260809 04:53:04.441326 21946 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [sys.catalog]: This master's current role is: LEADER
I20260809 04:53:04.445847 21961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:53:04.454192 21961 catalog_manager.cc:1347] Generated new cluster ID: 3bbd1aa07c9a496a83c2d5d472f47132
I20260809 04:53:04.454311 21961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:53:04.479590 21961 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:53:04.480890 21961 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:53:04.495394 21961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c: Generated new TSK 0
I20260809 04:53:04.496137 21961 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:53:04.557753 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44725
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:53:04.876480 21970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:04.876725 21970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:04.876814 21970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:04.883615 21970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:04.883702 21970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:04.883813 21970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:53:04.892894 21970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:04.893996 21970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:04.895238 21970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:04.907159 21977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:04.907207 21980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:04.907234 21978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:04.908160 21970 server_base.cc:1034] running on GCE node
I20260809 04:53:04.908715 21970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:04.909548 21970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:04.910749 21970 hybrid_clock.cc:648] HybridClock initialized: now 1786251184910710 us; error 27 us; skew 500 ppm
I20260809 04:53:04.911031 21970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:04 dist-test-slave-4tdj krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251184, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:04.913972 21970 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:53:04.915310 21970 webserver.cc:460] Webserver started at http://127.31.93.1:33539/ using document root <none> and password file <none>
I20260809 04:53:04.915940 21970 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:04.916056 21970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:04.916330 21970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:53:04.919137 21970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "c564c786e65e40cb8b28b23f94102673"
format_stamp: "Formatted at 2026-08-09 04:53:04 on dist-test-slave-4tdj"
server_key: "82176f520017a9148bc2b32ad5abf7f1"
server_key_iv: "6a1c013c7c7bf38393d36ebf3c97e826"
server_key_version: "encryptionkey@0"
I20260809 04:53:04.919911 21970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "c564c786e65e40cb8b28b23f94102673"
format_stamp: "Formatted at 2026-08-09 04:53:04 on dist-test-slave-4tdj"
server_key: "82176f520017a9148bc2b32ad5abf7f1"
server_key_iv: "6a1c013c7c7bf38393d36ebf3c97e826"
server_key_version: "encryptionkey@0"
I20260809 04:53:04.924590 21970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:53:04.927983 21990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:04.929216 21970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 04:53:04.929358 21970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "c564c786e65e40cb8b28b23f94102673"
format_stamp: "Formatted at 2026-08-09 04:53:04 on dist-test-slave-4tdj"
server_key: "82176f520017a9148bc2b32ad5abf7f1"
server_key_iv: "6a1c013c7c7bf38393d36ebf3c97e826"
server_key_version: "encryptionkey@0"
I20260809 04:53:04.929525 21970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:04.955826 21970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:04.959746 21970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:04.959990 21970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:04.960824 21970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:53:04.962023 21970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:53:04.962095 21970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:04.962215 21970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:53:04.962267 21970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:04.992015 21970 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:39573
I20260809 04:53:04.992087 22141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:39573 every 8 connection(s)
I20260809 04:53:04.993206 21970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:53:04.999336 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 21970
I20260809 04:53:04.999517 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:53:05.000181 32116 external_mini_cluster.cc:1428] Setting key a83d45782a3d833ea1e89900ff81dddb
Aug 09 04:53:04 dist-test-slave-4tdj krb5kdc[21824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251184, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:53:05.012171 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:04.995647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:45529 (local address 127.31.93.62:44725)
0809 04:53:04.996061 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:04.996065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:04.996892 (+ 827us) server_negotiation.cc:407] Connection header received
0809 04:53:04.998657 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:04.998663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:04.998751 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:04.998846 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:05.000434 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:05.001276 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:05.003011 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:05.003389 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:05.003506 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:05.006792 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:05.006796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:05.006800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:05.006834 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:05.009229 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:05.009897 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:05.009901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:05.009904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:05.010024 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:05.010472 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:05.010475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:05.010477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:05.010752 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:05.010885 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:05.011562 (+ 677us) server_negotiation.cc:299] Negotiation successful
0809 04:53:05.011792 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":204,"threads_started":1}
I20260809 04:53:05.013015 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:04.996062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:45529)
0809 04:53:04.996747 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:04.996785 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:04.998400 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:04.999091 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:04.999099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:04.999652 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:53:05.000265 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:05.000290 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:05.001439 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:05.001443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:05.002862 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:05.002876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:05.003613 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:05.003620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:05.003839 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:05.004450 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:53:05.004472 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:53:05.006588 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:53:05.009360 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:05.009369 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:05.009384 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:05.009775 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:05.010127 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:05.010132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:05.010135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:05.010355 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:05.010878 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:53:05.010885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:53:05.011197 (+ 312us) client_negotiation.cc:769] Sending connection context
0809 04:53:05.011419 (+ 222us) client_negotiation.cc:240] Negotiation successful
0809 04:53:05.011891 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":263,"threads_started":1}
I20260809 04:53:05.014618 22142 heartbeater.cc:347] Connected to a master server at 127.31.93.62:44725
I20260809 04:53:05.014923 22142 heartbeater.cc:464] Registering TS with master...
I20260809 04:53:05.015740 22142 heartbeater.cc:511] Master 127.31.93.62:44725 requested a full tablet report, sending...
I20260809 04:53:05.018049 21884 ts_manager.cc:194] Registered new tserver with Master: c564c786e65e40cb8b28b23f94102673 (127.31.93.1:39573)
I20260809 04:53:05.019972 21884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:45529
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:53:05.055665 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44725
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:53:05.365968 22150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:05.366218 22150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:05.366355 22150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:05.373276 22150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:05.373366 22150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:05.373503 22150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:53:05.382463 22150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:05.383565 22150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:05.384815 22150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:05.396819 22157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:05.396848 22158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:05.397862 22162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:05.397967 22150 server_base.cc:1034] running on GCE node
I20260809 04:53:05.398589 22150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:05.399423 22150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:05.400619 22150 hybrid_clock.cc:648] HybridClock initialized: now 1786251185400570 us; error 44 us; skew 500 ppm
I20260809 04:53:05.400897 22150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:05 dist-test-slave-4tdj krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251185, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:05.403854 22150 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:53:05.405117 22150 webserver.cc:460] Webserver started at http://127.31.93.2:32931/ using document root <none> and password file <none>
I20260809 04:53:05.405764 22150 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:05.405877 22150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:05.406157 22150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:53:05.409318 22150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "a84edde35edf48e9ad12018132c550b4"
format_stamp: "Formatted at 2026-08-09 04:53:05 on dist-test-slave-4tdj"
server_key: "48406ead1b40a66c3f1977775faec31b"
server_key_iv: "a9c81c73cfc86493d447e61fda55863b"
server_key_version: "encryptionkey@0"
I20260809 04:53:05.410195 22150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "a84edde35edf48e9ad12018132c550b4"
format_stamp: "Formatted at 2026-08-09 04:53:05 on dist-test-slave-4tdj"
server_key: "48406ead1b40a66c3f1977775faec31b"
server_key_iv: "a9c81c73cfc86493d447e61fda55863b"
server_key_version: "encryptionkey@0"
I20260809 04:53:05.415102 22150 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 04:53:05.418373 22177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:05.419683 22150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 04:53:05.419816 22150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "a84edde35edf48e9ad12018132c550b4"
format_stamp: "Formatted at 2026-08-09 04:53:05 on dist-test-slave-4tdj"
server_key: "48406ead1b40a66c3f1977775faec31b"
server_key_iv: "a9c81c73cfc86493d447e61fda55863b"
server_key_version: "encryptionkey@0"
I20260809 04:53:05.419983 22150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:05.432904 22150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:05.436873 22150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:05.437119 22150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:05.438009 22150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:53:05.439277 22150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:53:05.439350 22150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:05.439436 22150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:53:05.439481 22150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:05.469226 22150 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:46529
I20260809 04:53:05.469290 22331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:46529 every 8 connection(s)
I20260809 04:53:05.470432 22150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:53:05.476574 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 22150
I20260809 04:53:05.476727 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:53:05.477144 32116 external_mini_cluster.cc:1428] Setting key 626a4487316a8c4615335d5d7584e931
Aug 09 04:53:05 dist-test-slave-4tdj krb5kdc[21824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251185, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:53:05.489022 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:05.472786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:49491 (local address 127.31.93.62:44725)
0809 04:53:05.472966 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:05.472970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:05.474154 (+ 1184us) server_negotiation.cc:407] Connection header received
0809 04:53:05.475898 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:05.475902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:05.475972 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:05.476068 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:05.477547 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:05.478361 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:05.479845 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:05.480254 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:05.480360 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:05.483551 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:05.483557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:05.483560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:05.483591 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:05.486079 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:05.486810 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:05.486816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:05.486818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:05.486951 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:05.487439 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:05.487442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:05.487444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:05.487723 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:05.487846 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:05.488526 (+ 680us) server_negotiation.cc:299] Negotiation successful
0809 04:53:05.488755 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 04:53:05.489998 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:05.473257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:49491)
0809 04:53:05.474003 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:05.474041 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:05.475644 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:05.476252 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:05.476260 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:05.476810 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:53:05.477380 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:05.477398 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:05.478509 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:05.478513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:05.479711 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:05.479724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:05.480459 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:05.480467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:05.480680 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:05.481287 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:53:05.481309 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:53:05.483350 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:53:05.486228 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:05.486237 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:05.486253 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:05.486696 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:05.487090 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:05.487094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:05.487098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:05.487336 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:05.487838 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:53:05.487845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:53:05.488203 (+ 358us) client_negotiation.cc:769] Sending connection context
0809 04:53:05.488460 (+ 257us) client_negotiation.cc:240] Negotiation successful
0809 04:53:05.488937 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":310,"threads_started":1}
I20260809 04:53:05.491304 22332 heartbeater.cc:347] Connected to a master server at 127.31.93.62:44725
I20260809 04:53:05.491629 22332 heartbeater.cc:464] Registering TS with master...
I20260809 04:53:05.492419 22332 heartbeater.cc:511] Master 127.31.93.62:44725 requested a full tablet report, sending...
I20260809 04:53:05.493916 21884 ts_manager.cc:194] Registered new tserver with Master: a84edde35edf48e9ad12018132c550b4 (127.31.93.2:46529)
I20260809 04:53:05.494767 21884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:49491
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:53:05.530930 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44725
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:53:05.845113 22339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:05.845397 22339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:05.845536 22339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:05.852414 22339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:05.852526 22339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:05.852676 22339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:53:05.861825 22339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:05.863123 22339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:05.864373 22339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:05.876523 22345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:05.876493 22348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:05.876551 22346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:05.877552 22339 server_base.cc:1034] running on GCE node
I20260809 04:53:05.878114 22339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:05.879012 22339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:05.880208 22339 hybrid_clock.cc:648] HybridClock initialized: now 1786251185880122 us; error 95 us; skew 500 ppm
I20260809 04:53:05.880505 22339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:05 dist-test-slave-4tdj krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251185, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:05.883411 22339 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:53:05.884727 22339 webserver.cc:460] Webserver started at http://127.31.93.3:34873/ using document root <none> and password file <none>
I20260809 04:53:05.885377 22339 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:05.885495 22339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:05.885792 22339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:53:05.888788 22339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "a8ff1545118c4085883aedeec0da494d"
format_stamp: "Formatted at 2026-08-09 04:53:05 on dist-test-slave-4tdj"
server_key: "ae43678b8ad672020036812069fe655e"
server_key_iv: "53feb349480fd92f9af566964c586098"
server_key_version: "encryptionkey@0"
I20260809 04:53:05.889597 22339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "a8ff1545118c4085883aedeec0da494d"
format_stamp: "Formatted at 2026-08-09 04:53:05 on dist-test-slave-4tdj"
server_key: "ae43678b8ad672020036812069fe655e"
server_key_iv: "53feb349480fd92f9af566964c586098"
server_key_version: "encryptionkey@0"
I20260809 04:53:05.894470 22339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 04:53:05.897716 22361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:05.899034 22339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:53:05.899196 22339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "a8ff1545118c4085883aedeec0da494d"
format_stamp: "Formatted at 2026-08-09 04:53:05 on dist-test-slave-4tdj"
server_key: "ae43678b8ad672020036812069fe655e"
server_key_iv: "53feb349480fd92f9af566964c586098"
server_key_version: "encryptionkey@0"
I20260809 04:53:05.899370 22339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:05.930514 22339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:05.934056 22339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:05.934352 22339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:05.935333 22339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:53:05.936609 22339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:53:05.936687 22339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:05.936800 22339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:53:05.936856 22339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:05.966248 22339 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:34783
I20260809 04:53:05.966308 22518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:34783 every 8 connection(s)
I20260809 04:53:05.967499 22339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:53:05.970724 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 22339
I20260809 04:53:05.970918 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:53:05.971474 32116 external_mini_cluster.cc:1428] Setting key 84694da1a0fc58282a1cab0a43d44f74
Aug 09 04:53:05 dist-test-slave-4tdj krb5kdc[21824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251185, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:53:05.986181 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:05.969831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:34469 (local address 127.31.93.62:44725)
0809 04:53:05.970003 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:05.970007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:05.971346 (+ 1339us) server_negotiation.cc:407] Connection header received
0809 04:53:05.973405 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:05.973411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:05.973494 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:05.973611 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:05.975213 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:05.975930 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:05.977429 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:05.977844 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:05.977951 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:05.981076 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:05.981081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:05.981084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:05.981117 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:05.983350 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:05.984000 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:05.984004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:05.984006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:05.984126 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:05.984520 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:05.984523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:05.984525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:05.984804 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:05.984935 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:05.985577 (+ 642us) server_negotiation.cc:299] Negotiation successful
0809 04:53:05.985830 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 04:53:05.987043 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:05.970383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:34469)
0809 04:53:05.971197 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:05.971241 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:05.973107 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:05.973841 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:05.973849 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:05.974499 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:53:05.975047 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:05.975063 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:05.976077 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:05.976081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:05.977289 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:05.977301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:05.978067 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:05.978073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:05.978322 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:05.978883 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:53:05.978909 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:53:05.980885 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:53:05.983477 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:05.983486 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:05.983501 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:05.983879 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:05.984231 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:05.984236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:05.984239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:05.984407 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:05.984926 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:53:05.984933 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:53:05.985240 (+ 307us) client_negotiation.cc:769] Sending connection context
0809 04:53:05.985464 (+ 224us) client_negotiation.cc:240] Negotiation successful
0809 04:53:05.985943 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":341,"threads_started":1}
I20260809 04:53:05.988224 22521 heartbeater.cc:347] Connected to a master server at 127.31.93.62:44725
I20260809 04:53:05.988507 22521 heartbeater.cc:464] Registering TS with master...
I20260809 04:53:05.989188 22521 heartbeater.cc:511] Master 127.31.93.62:44725 requested a full tablet report, sending...
I20260809 04:53:05.990617 21884 ts_manager.cc:194] Registered new tserver with Master: a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783)
I20260809 04:53:05.991381 21884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:34469
I20260809 04:53:05.998191 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:53:06.013270 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:06.002400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44256 (local address 127.31.93.62:44725)
0809 04:53:06.002579 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:06.002585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:06.002947 (+ 362us) server_negotiation.cc:407] Connection header received
0809 04:53:06.003068 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:06.003073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:06.003150 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:06.003233 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:06.004514 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.005199 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.006416 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.006792 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.006875 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:06.008958 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:06.008963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:06.008966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:06.008989 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:06.011200 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:06.011782 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:06.011786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:06.011788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:06.011910 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:06.012200 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:06.012202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:06.012204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:06.012475 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:06.012647 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:06.012873 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 04:53:06.013020 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:53:06.016979 21884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 04:53:06.023352 22142 heartbeater.cc:503] Master 127.31.93.62:44725 was elected leader, sending a full tablet report...
W20260809 04:53:06.023815 21884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:53:06.050931 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:06.035487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.31.93.1:39573)
0809 04:53:06.043646 (+ 8159us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:06.043656 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:06.043687 (+ 31us) server_negotiation.cc:407] Connection header received
0809 04:53:06.043787 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:06.043795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:06.044067 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:06.044244 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:06.044868 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.046519 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.048627 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.049718 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.049832 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:06.050067 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:06.050431 (+ 364us) server_negotiation.cc:299] Negotiation successful
0809 04:53:06.050610 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7985,"thread_start_us":7822,"threads_started":1}
I20260809 04:53:06.051020 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:06.035233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:39573 (local address 127.0.0.1:54420)
0809 04:53:06.036067 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:06.036112 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:06.036294 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:06.044259 (+ 7965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:06.044266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:06.044304 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:06.044643 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:06.044657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.046674 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.046680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:06.048470 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:06.048484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.049916 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.049923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:06.050195 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:06.050235 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 04:53:06.050384 (+ 149us) client_negotiation.cc:240] Negotiation successful
0809 04:53:06.050589 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":35,"thread_start_us":997,"threads_started":1}
I20260809 04:53:06.051761 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:06.035921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:34783 (local address 127.0.0.1:36308)
0809 04:53:06.036910 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:06.036928 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:06.037074 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:06.044689 (+ 7615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:06.044694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:06.044717 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:06.045114 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:06.045127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.047051 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.047057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:06.048946 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:06.048960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.050973 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.050980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:06.051156 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:06.051185 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 04:53:06.051263 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 04:53:06.051375 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":1,"thread_start_us":911,"threads_started":1}
I20260809 04:53:06.052564 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:06.036607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36308 (local address 127.31.93.3:34783)
0809 04:53:06.044253 (+ 7646us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:06.044258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:06.044282 (+ 24us) server_negotiation.cc:407] Connection header received
0809 04:53:06.044353 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:06.044358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:06.044515 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:06.044674 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:06.045325 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.046905 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.049124 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.050495 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.050609 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:06.050817 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:06.052053 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0809 04:53:06.052221 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7497,"thread_start_us":7363,"threads_started":1}
I20260809 04:53:06.053387 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:06.036999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:46529 (local address 127.0.0.1:48454)
0809 04:53:06.043877 (+ 6878us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:06.043894 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:06.043994 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:06.045718 (+ 1724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:06.045722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:06.045744 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:06.046099 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:06.046112 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.048138 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.048143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:06.050180 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:06.050196 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.052651 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.052658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:06.052813 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:06.052843 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 04:53:06.052935 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 04:53:06.053049 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6771,"mutex_wait_us":49,"thread_start_us":6636,"threads_started":1}
I20260809 04:53:06.053485 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:06.037498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.31.93.2:46529)
0809 04:53:06.045299 (+ 7801us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:06.045304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:06.045325 (+ 21us) server_negotiation.cc:407] Connection header received
0809 04:53:06.045393 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:06.045397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:06.045556 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:06.045705 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:06.046634 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.047992 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.050351 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.052207 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.052336 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:06.052515 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:06.053014 (+ 499us) server_negotiation.cc:299] Negotiation successful
0809 04:53:06.053179 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7637,"thread_start_us":7514,"threads_started":1}
I20260809 04:53:06.055156 22053 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa9065eaec24bc385470a7668703427 (DEFAULT_TABLE table=test-table [id=38401296705e4785ad15b18f61ae212f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:53:06.056376 22053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa9065eaec24bc385470a7668703427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:53:06.055968 22426 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa9065eaec24bc385470a7668703427 (DEFAULT_TABLE table=test-table [id=38401296705e4785ad15b18f61ae212f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:53:06.057385 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa9065eaec24bc385470a7668703427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:53:06.056910 22244 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa9065eaec24bc385470a7668703427 (DEFAULT_TABLE table=test-table [id=38401296705e4785ad15b18f61ae212f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:53:06.058355 22244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa9065eaec24bc385470a7668703427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:53:06.067315 22549 tablet_bootstrap.cc:492] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673: Bootstrap starting.
I20260809 04:53:06.070120 22549 tablet_bootstrap.cc:654] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673: Neither blocks nor log segments found. Creating new log.
I20260809 04:53:06.071261 22550 tablet_bootstrap.cc:492] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d: Bootstrap starting.
I20260809 04:53:06.071338 22549 log.cc:824] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:06.074939 22550 tablet_bootstrap.cc:654] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d: Neither blocks nor log segments found. Creating new log.
I20260809 04:53:06.075618 22551 tablet_bootstrap.cc:492] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4: Bootstrap starting.
I20260809 04:53:06.076233 22550 log.cc:824] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:06.077170 22549 tablet_bootstrap.cc:492] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673: No bootstrap required, opened a new log
I20260809 04:53:06.077447 22549 ts_tablet_manager.cc:1397] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260809 04:53:06.078860 22551 tablet_bootstrap.cc:654] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4: Neither blocks nor log segments found. Creating new log.
I20260809 04:53:06.079238 22550 tablet_bootstrap.cc:492] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d: No bootstrap required, opened a new log
I20260809 04:53:06.079488 22550 ts_tablet_manager.cc:1397] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 04:53:06.080006 22551 log.cc:824] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:06.082782 22551 tablet_bootstrap.cc:492] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4: No bootstrap required, opened a new log
I20260809 04:53:06.083021 22551 ts_tablet_manager.cc:1397] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 04:53:06.084822 22549 raft_consensus.cc:348] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } }
I20260809 04:53:06.085180 22549 raft_consensus.cc:374] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:53:06.085294 22549 raft_consensus.cc:729] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c564c786e65e40cb8b28b23f94102673, State: Initialized, Role: FOLLOWER
I20260809 04:53:06.085834 22549 consensus_queue.cc:260] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } }
I20260809 04:53:06.087256 22549 ts_tablet_manager.cc:1428] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260809 04:53:06.089048 22550 raft_consensus.cc:348] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } }
I20260809 04:53:06.089421 22550 raft_consensus.cc:374] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:53:06.089516 22550 raft_consensus.cc:729] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ff1545118c4085883aedeec0da494d, State: Initialized, Role: FOLLOWER
I20260809 04:53:06.090096 22550 consensus_queue.cc:260] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } }
I20260809 04:53:06.091136 22521 heartbeater.cc:503] Master 127.31.93.62:44725 was elected leader, sending a full tablet report...
I20260809 04:53:06.091691 22550 ts_tablet_manager.cc:1428] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260809 04:53:06.093356 22551 raft_consensus.cc:348] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } }
I20260809 04:53:06.093667 22551 raft_consensus.cc:374] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:53:06.093752 22551 raft_consensus.cc:729] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84edde35edf48e9ad12018132c550b4, State: Initialized, Role: FOLLOWER
I20260809 04:53:06.094208 22551 consensus_queue.cc:260] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } }
I20260809 04:53:06.095233 22332 heartbeater.cc:503] Master 127.31.93.62:44725 was elected leader, sending a full tablet report...
I20260809 04:53:06.095647 22551 ts_tablet_manager.cc:1428] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260809 04:53:06.209808 22561 raft_consensus.cc:482] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:53:06.210121 22561 raft_consensus.cc:504] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } }
I20260809 04:53:06.211833 22561 leader_election.cc:288] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c564c786e65e40cb8b28b23f94102673 (127.31.93.1:39573), a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783)
I20260809 04:53:06.218194 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:06.212464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:40709 (local address 127.31.93.3:34783)
0809 04:53:06.212654 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:06.212659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:06.212961 (+ 302us) server_negotiation.cc:407] Connection header received
0809 04:53:06.213148 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:06.213153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:06.213226 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:06.213313 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:06.213895 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.214700 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.216513 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.217250 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.217338 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:06.217426 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:06.217834 (+ 408us) server_negotiation.cc:299] Negotiation successful
0809 04:53:06.217953 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 04:53:06.218201 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:06.212258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:56985 (local address 127.31.93.1:39573)
0809 04:53:06.212621 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:06.212628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:06.213167 (+ 539us) server_negotiation.cc:407] Connection header received
0809 04:53:06.213311 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:06.213315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:06.213389 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:06.213483 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:06.214061 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.214826 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.216533 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.217301 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.217437 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:06.217532 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:06.217832 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 04:53:06.217955 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260809 04:53:06.218400 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:06.212094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:34783 (local address 127.31.93.2:40709)
0809 04:53:06.212847 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:06.212868 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:06.213014 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:06.213334 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:06.213338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:06.213361 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:06.213750 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:06.213761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.214823 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.214828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:06.216367 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:06.216379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.217475 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.217481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:06.217663 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:06.217712 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 04:53:06.217789 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 04:53:06.217918 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":45,"thread_start_us":361,"threads_started":1}
I20260809 04:53:06.218446 22563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:06.212096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:39573 (local address 127.31.93.2:56985)
0809 04:53:06.213050 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:06.213071 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:06.213199 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:06.213514 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:06.213517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:06.213540 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:06.213912 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:06.213924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.214969 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.214973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:06.216386 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:06.216398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.217527 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.217532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:06.217667 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:06.217705 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 04:53:06.217780 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 04:53:06.217921 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":160,"thread_start_us":496,"threads_started":1}
W20260809 04:53:06.219054 22522 tablet.cc:2369] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:53:06.219545 22453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa9065eaec24bc385470a7668703427" candidate_uuid: "a84edde35edf48e9ad12018132c550b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ff1545118c4085883aedeec0da494d" is_pre_election: true
I20260809 04:53:06.219535 22083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa9065eaec24bc385470a7668703427" candidate_uuid: "a84edde35edf48e9ad12018132c550b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c564c786e65e40cb8b28b23f94102673" is_pre_election: true
I20260809 04:53:06.220110 22453 raft_consensus.cc:2454] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a84edde35edf48e9ad12018132c550b4 in term 0.
I20260809 04:53:06.220100 22083 raft_consensus.cc:2454] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a84edde35edf48e9ad12018132c550b4 in term 0.
I20260809 04:53:06.220755 22181 leader_election.cc:302] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a84edde35edf48e9ad12018132c550b4, c564c786e65e40cb8b28b23f94102673; no voters:
I20260809 04:53:06.221127 22561 raft_consensus.cc:2781] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:53:06.221279 22561 raft_consensus.cc:482] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:53:06.221385 22561 raft_consensus.cc:3037] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [term 0 FOLLOWER]: Advancing to term 1
W20260809 04:53:06.222209 22333 tablet.cc:2369] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:53:06.224086 22561 raft_consensus.cc:504] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } }
I20260809 04:53:06.224645 22561 leader_election.cc:288] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [CANDIDATE]: Term 1 election: Requested vote from peers c564c786e65e40cb8b28b23f94102673 (127.31.93.1:39573), a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783)
I20260809 04:53:06.225056 22453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa9065eaec24bc385470a7668703427" candidate_uuid: "a84edde35edf48e9ad12018132c550b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ff1545118c4085883aedeec0da494d"
I20260809 04:53:06.225061 22083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa9065eaec24bc385470a7668703427" candidate_uuid: "a84edde35edf48e9ad12018132c550b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c564c786e65e40cb8b28b23f94102673"
I20260809 04:53:06.225229 22083 raft_consensus.cc:3037] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:53:06.225250 22453 raft_consensus.cc:3037] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:53:06.227365 22453 raft_consensus.cc:2454] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a84edde35edf48e9ad12018132c550b4 in term 1.
I20260809 04:53:06.227413 22083 raft_consensus.cc:2454] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a84edde35edf48e9ad12018132c550b4 in term 1.
I20260809 04:53:06.227731 22180 leader_election.cc:302] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a84edde35edf48e9ad12018132c550b4, a8ff1545118c4085883aedeec0da494d; no voters:
I20260809 04:53:06.228010 22561 raft_consensus.cc:2781] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:53:06.228394 22561 raft_consensus.cc:686] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [term 1 LEADER]: Becoming Leader. State: Replica: a84edde35edf48e9ad12018132c550b4, State: Running, Role: LEADER
I20260809 04:53:06.228820 22561 consensus_queue.cc:237] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } }
I20260809 04:53:06.233400 21883 catalog_manager.cc:5495] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a84edde35edf48e9ad12018132c550b4 (127.31.93.2). New cstate: current_term: 1 leader_uuid: "a84edde35edf48e9ad12018132c550b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } health_report { overall_health: HEALTHY } } }
W20260809 04:53:06.245347 22143 tablet.cc:2369] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:53:06.278769 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:06.273709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48462 (local address 127.31.93.2:46529)
0809 04:53:06.273928 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:06.273934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:06.273952 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:53:06.274040 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:06.274043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:06.274116 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:06.274245 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:53:06.274653 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.275375 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.276750 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.277141 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.277245 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:06.277710 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:53:06.278177 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:53:06.278338 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:06.278439 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 04:53:06.278514 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 04:53:06.286273 22453 raft_consensus.cc:1261] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d [term 1 FOLLOWER]: Refusing update from remote peer a84edde35edf48e9ad12018132c550b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:53:06.286273 22083 raft_consensus.cc:1261] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Refusing update from remote peer a84edde35edf48e9ad12018132c550b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:53:06.287158 22565 consensus_queue.cc:1035] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:53:06.287482 22561 consensus_queue.cc:1035] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:53:06.301189 22570 mvcc.cc:205] Tried to move back new op lower bound from 7316484859019325440 to 7316484858796924928. Current Snapshot: MvccSnapshot[applied={T|T < 7316484859019325440}]
I20260809 04:53:06.310788 21883 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:53:06.311094 21883 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:53:06.314349 21883 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c: Sending DeleteTablet for 3 replicas of tablet 9aa9065eaec24bc385470a7668703427
I20260809 04:53:06.315446 22426 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa9065eaec24bc385470a7668703427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:53:06 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:36308
I20260809 04:53:06.315683 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 21970
I20260809 04:53:06.315750 22244 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa9065eaec24bc385470a7668703427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:53:06 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:48454
I20260809 04:53:06.316143 22053 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa9065eaec24bc385470a7668703427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:53:06 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:54420
I20260809 04:53:06.326985 22590 tablet_replica.cc:330] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673: stopping tablet replica
I20260809 04:53:06.328262 22587 tablet_replica.cc:330] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d: stopping tablet replica
I20260809 04:53:06.328709 22587 raft_consensus.cc:2229] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:53:06.329232 22587 raft_consensus.cc:2258] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:53:06.327538 22589 tablet_replica.cc:330] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4: stopping tablet replica
I20260809 04:53:06.329905 22589 raft_consensus.cc:2229] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:53:06.330554 22589 raft_consensus.cc:2258] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:53:06.331166 22590 raft_consensus.cc:2229] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:53:06.331614 22590 raft_consensus.cc:2258] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:53:06.333012 22589 ts_tablet_manager.cc:1905] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:53:06.334061 22587 ts_tablet_manager.cc:1905] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:53:06.337868 22590 ts_tablet_manager.cc:1905] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:53:06.340910 22587 ts_tablet_manager.cc:1918] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:53:06.341068 22587 log.cc:1196] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/9aa9065eaec24bc385470a7668703427
I20260809 04:53:06.341632 22587 ts_tablet_manager.cc:1939] T 9aa9065eaec24bc385470a7668703427 P a8ff1545118c4085883aedeec0da494d: Deleting consensus metadata
Aug 09 04:53:06 dist-test-slave-4tdj krb5kdc[21824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:53:06.343436 21864 catalog_manager.cc:4841] TS a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783): tablet 9aa9065eaec24bc385470a7668703427 (table test-table [id=38401296705e4785ad15b18f61ae212f]) successfully deleted
I20260809 04:53:06.344262 22590 ts_tablet_manager.cc:1918] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:53:06.344449 22590 log.cc:1196] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/9aa9065eaec24bc385470a7668703427
I20260809 04:53:06.344993 22590 ts_tablet_manager.cc:1939] T 9aa9065eaec24bc385470a7668703427 P c564c786e65e40cb8b28b23f94102673: Deleting consensus metadata
I20260809 04:53:06.345640 22589 ts_tablet_manager.cc:1918] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:53:06.345825 22589 log.cc:1196] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/9aa9065eaec24bc385470a7668703427
I20260809 04:53:06.346393 22589 ts_tablet_manager.cc:1939] T 9aa9065eaec24bc385470a7668703427 P a84edde35edf48e9ad12018132c550b4: Deleting consensus metadata
I20260809 04:53:06.346719 21865 catalog_manager.cc:4841] TS c564c786e65e40cb8b28b23f94102673 (127.31.93.1:39573): tablet 9aa9065eaec24bc385470a7668703427 (table test-table [id=38401296705e4785ad15b18f61ae212f]) successfully deleted
I20260809 04:53:06.347976 21865 catalog_manager.cc:4841] TS a84edde35edf48e9ad12018132c550b4 (127.31.93.2:46529): tablet 9aa9065eaec24bc385470a7668703427 (table test-table [id=38401296705e4785ad15b18f61ae212f]) successfully deleted
I20260809 04:53:06.350564 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:06.316869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54436 (local address 127.31.93.1:39573)
0809 04:53:06.317058 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:06.317064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:06.332471 (+ 15407us) server_negotiation.cc:407] Connection header received
0809 04:53:06.332689 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:06.332695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:06.332778 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:06.332869 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:06.335145 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.336295 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.338230 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.338760 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.338873 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:06.344311 (+ 5438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:06.344324 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:06.344340 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:06.344379 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:06.347868 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:06.348569 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:06.348577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:06.348580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:06.348705 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:06.349086 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:06.349090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:06.349092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:06.349409 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:06.349695 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:06.349980 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 04:53:06.350197 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:53:06.351110 22137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:06.440220 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 21970
I20260809 04:53:06.465947 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 22150
Aug 09 04:53:06 dist-test-slave-4tdj krb5kdc[21824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:53:06.485405 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:06.466896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48468 (local address 127.31.93.2:46529)
0809 04:53:06.467049 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:06.467053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:06.467070 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:53:06.467217 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:06.467221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:06.467283 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:06.467374 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:06.469034 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.470244 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.472203 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.472897 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.473020 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:06.478510 (+ 5490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:06.478523 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:06.478543 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:06.478598 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:06.482426 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:06.483206 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:06.483215 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:06.483219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:06.483397 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:06.483806 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:06.483811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:06.483814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:06.484225 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:06.484522 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:06.484819 (+ 297us) server_negotiation.cc:299] Negotiation successful
0809 04:53:06.485031 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 04:53:06.486148 22326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:06.594502 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 22150
I20260809 04:53:06.614627 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 22339
Aug 09 04:53:06 dist-test-slave-4tdj krb5kdc[21824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:53:06.629379 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:06.615364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.31.93.3:34783)
0809 04:53:06.615547 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:06.615553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:06.615624 (+ 71us) server_negotiation.cc:407] Connection header received
0809 04:53:06.615806 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:06.615811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:06.615883 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:06.615980 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:06.617266 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.617979 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.619354 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:06.619758 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:06.619871 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:06.624009 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:06.624022 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:06.624039 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:06.624072 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:06.626749 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:06.627382 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:06.627389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:06.627391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:06.627532 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:06.627898 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:06.627902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:06.627904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:06.628249 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:06.628540 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:06.628845 (+ 305us) server_negotiation.cc:299] Negotiation successful
0809 04:53:06.629054 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 04:53:06.629977 22511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:06.715248 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 22339
I20260809 04:53:06.738725 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 21841
I20260809 04:53:06.739807 21926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:06.848340 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 21841
I20260809 04:53:06.872538 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:44725
--webserver_interface=127.31.93.62
--webserver_port=33647
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:44725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:53:07.196653 22600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:07.196913 22600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:07.197038 22600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:07.204178 22600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:53:07.204281 22600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:07.204355 22600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:07.204425 22600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:53:07.204483 22600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:53:07.214008 22600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:44725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:44725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=33647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:07.215248 22600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:07.216531 22600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:07.241174 22609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:07.241195 22607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:07.241222 22614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:07.241914 22600 server_base.cc:1034] running on GCE node
I20260809 04:53:07.242774 22600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:07.244275 22600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:07.245472 22600 hybrid_clock.cc:648] HybridClock initialized: now 1786251187245423 us; error 38 us; skew 500 ppm
I20260809 04:53:07.245855 22600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:07 dist-test-slave-4tdj krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251187, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:07.249265 22600 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:53:07.250933 22600 webserver.cc:460] Webserver started at http://127.31.93.62:33647/ using document root <none> and password file <none>
I20260809 04:53:07.251653 22600 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:07.251784 22600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:07.257944 22600 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:53:07.260797 22621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:07.262043 22600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 04:53:07.262209 22600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c"
format_stamp: "Formatted at 2026-08-09 04:53:04 on dist-test-slave-4tdj"
server_key: "57ea955e214b39468e3982c4fdca4242"
server_key_iv: "2606f6902ae7924d4097eca2d8d50fc2"
server_key_version: "encryptionkey@0"
I20260809 04:53:07.263289 22600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:07.281497 22600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:07.285507 22600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:07.285780 22600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:07.303587 22600 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:44725
I20260809 04:53:07.303609 22697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:44725 every 8 connection(s)
I20260809 04:53:07.305049 22600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:53:07.312103 22698 sys_catalog.cc:263] Verifying existing consensus state
I20260809 04:53:07.314893 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 22600
I20260809 04:53:07.315899 22698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c: Bootstrap starting.
I20260809 04:53:07.315975 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:39573
--local_ip_for_outbound_sockets=127.31.93.1
--tserver_master_addrs=127.31.93.62:44725
--webserver_port=33539
--webserver_interface=127.31.93.1
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:53:07.337772 22698 log.cc:824] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:07.349969 22698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:53:07.350512 22698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c: Bootstrap complete.
I20260809 04:53:07.359480 22698 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } }
I20260809 04:53:07.360288 22698 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd94ed2317924c2c9c1dfd6caf15aa3c, State: Initialized, Role: FOLLOWER
I20260809 04:53:07.360904 22698 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } }
I20260809 04:53:07.361137 22698 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:53:07.361251 22698 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:53:07.361404 22698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 1 FOLLOWER]: Advancing to term 2
I20260809 04:53:07.365135 22698 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } }
I20260809 04:53:07.365645 22698 leader_election.cc:302] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd94ed2317924c2c9c1dfd6caf15aa3c; no voters:
I20260809 04:53:07.366235 22698 leader_election.cc:288] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 04:53:07.366412 22704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 2 FOLLOWER]: Leader election won for term 2
I20260809 04:53:07.367744 22704 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 2 LEADER]: Becoming Leader. State: Replica: fd94ed2317924c2c9c1dfd6caf15aa3c, State: Running, Role: LEADER
I20260809 04:53:07.368595 22698 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:53:07.368471 22704 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } }
I20260809 04:53:07.370970 22705 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } } }
I20260809 04:53:07.371366 22705 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [sys.catalog]: This master's current role is: LEADER
I20260809 04:53:07.372705 22706 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd94ed2317924c2c9c1dfd6caf15aa3c. Latest consensus state: current_term: 2 leader_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } } }
I20260809 04:53:07.373044 22706 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [sys.catalog]: This master's current role is: LEADER
I20260809 04:53:07.374241 22709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:53:07.381981 22709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:53:07.383464 22709 catalog_manager.cc:1259] Loaded cluster ID: 3bbd1aa07c9a496a83c2d5d472f47132
I20260809 04:53:07.383622 22709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:53:07.386816 22709 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260809 04:53:07.404608 22709 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260809 04:53:07.426522 22709 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:53:07.428222 22709 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c: Loaded TSK: 0
I20260809 04:53:07.429783 22709 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 04:53:07.664336 22701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:07.664642 22701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:07.664768 22701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:07.672113 22701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:07.672214 22701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:07.672356 22701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:53:07.681993 22701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:39573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=33539
--tserver_master_addrs=127.31.93.62:44725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:07.683311 22701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:07.684612 22701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:07.698711 22739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:07.698678 22736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:07.699216 22735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:07.700001 22701 server_base.cc:1034] running on GCE node
I20260809 04:53:07.700637 22701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:07.701596 22701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:07.702798 22701 hybrid_clock.cc:648] HybridClock initialized: now 1786251187702721 us; error 71 us; skew 500 ppm
I20260809 04:53:07.703086 22701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:07 dist-test-slave-4tdj krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251187, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:07.706171 22701 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:53:07.707640 22701 webserver.cc:460] Webserver started at http://127.31.93.1:33539/ using document root <none> and password file <none>
I20260809 04:53:07.708281 22701 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:07.708412 22701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:07.713784 22701 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:53:07.716496 22749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:07.717789 22701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:53:07.717932 22701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "c564c786e65e40cb8b28b23f94102673"
format_stamp: "Formatted at 2026-08-09 04:53:04 on dist-test-slave-4tdj"
server_key: "82176f520017a9148bc2b32ad5abf7f1"
server_key_iv: "6a1c013c7c7bf38393d36ebf3c97e826"
server_key_version: "encryptionkey@0"
I20260809 04:53:07.718897 22701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:07.752230 22701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:07.756354 22701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:07.756603 22701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:07.757542 22701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:53:07.758849 22701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:53:07.758934 22701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:07.759039 22701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:53:07.759111 22701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:07.789537 22701 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:39573
I20260809 04:53:07.789606 22919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:39573 every 8 connection(s)
I20260809 04:53:07.790927 22701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:53:07.793850 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 22701
I20260809 04:53:07.794661 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:46529
--local_ip_for_outbound_sockets=127.31.93.2
--tserver_master_addrs=127.31.93.62:44725
--webserver_port=32931
--webserver_interface=127.31.93.2
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:53:07 dist-test-slave-4tdj krb5kdc[21824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251187, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:53:07.815227 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:07.793893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:55151)
0809 04:53:07.794744 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:07.794801 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:07.796498 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:07.797907 (+ 1409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:07.797918 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:07.798571 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:53:07.799106 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:07.799124 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:07.801070 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:07.801075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:07.802704 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:07.802719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:07.804104 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:07.804113 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:07.804350 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:07.804980 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:53:07.805016 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:53:07.807430 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:53:07.811617 (+ 4187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:07.811624 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:07.811633 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:07.812008 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:07.812499 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:07.812502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:07.812504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:07.812671 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:07.813513 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:53:07.813518 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:53:07.813784 (+ 266us) client_negotiation.cc:769] Sending connection context
0809 04:53:07.813980 (+ 196us) client_negotiation.cc:240] Negotiation successful
0809 04:53:07.814331 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":366,"threads_started":1}
I20260809 04:53:07.815924 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:07.793630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:55151 (local address 127.31.93.62:44725)
0809 04:53:07.794276 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:07.794316 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:07.796792 (+ 2476us) server_negotiation.cc:407] Connection header received
0809 04:53:07.797088 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:07.797098 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:07.797396 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:07.797735 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:07.799338 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:07.800856 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:07.802875 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:07.803800 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:07.803978 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:07.807730 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:07.807741 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:07.807758 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:07.807804 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:07.811482 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:07.812167 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:07.812176 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:07.812184 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:07.812399 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:07.812788 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:07.812793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:07.812796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:07.813396 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:07.813617 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:07.814267 (+ 650us) server_negotiation.cc:299] Negotiation successful
0809 04:53:07.814676 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":251,"threads_started":1}
I20260809 04:53:07.818600 22920 heartbeater.cc:347] Connected to a master server at 127.31.93.62:44725
I20260809 04:53:07.818938 22920 heartbeater.cc:464] Registering TS with master...
I20260809 04:53:07.819854 22920 heartbeater.cc:511] Master 127.31.93.62:44725 requested a full tablet report, sending...
I20260809 04:53:07.822197 22642 ts_manager.cc:194] Registered new tserver with Master: c564c786e65e40cb8b28b23f94102673 (127.31.93.1:39573)
I20260809 04:53:07.823741 22642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:55151
W20260809 04:53:08.116449 22927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:08.116709 22927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:08.116842 22927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:08.123972 22927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:08.124086 22927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:08.124231 22927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:53:08.135633 22927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:46529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=32931
--tserver_master_addrs=127.31.93.62:44725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:08.137259 22927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:08.138586 22927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:08.152498 22933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:08.152463 22934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:08.153144 22937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:08.153514 22927 server_base.cc:1034] running on GCE node
I20260809 04:53:08.154080 22927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:08.155046 22927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:08.156255 22927 hybrid_clock.cc:648] HybridClock initialized: now 1786251188156211 us; error 38 us; skew 500 ppm
I20260809 04:53:08.156553 22927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:08 dist-test-slave-4tdj krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251188, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:08.159708 22927 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:53:08.161237 22927 webserver.cc:460] Webserver started at http://127.31.93.2:32931/ using document root <none> and password file <none>
I20260809 04:53:08.161872 22927 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:08.162001 22927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:08.167184 22927 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 04:53:08.169868 22947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:08.171226 22927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:53:08.171375 22927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "a84edde35edf48e9ad12018132c550b4"
format_stamp: "Formatted at 2026-08-09 04:53:05 on dist-test-slave-4tdj"
server_key: "48406ead1b40a66c3f1977775faec31b"
server_key_iv: "a9c81c73cfc86493d447e61fda55863b"
server_key_version: "encryptionkey@0"
I20260809 04:53:08.172281 22927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:08.203401 22927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:08.207497 22927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:08.207736 22927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:08.208684 22927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:53:08.209985 22927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:53:08.210088 22927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:08.210191 22927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:53:08.210260 22927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:08.246806 22927 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:46529
I20260809 04:53:08.246922 23107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:46529 every 8 connection(s)
I20260809 04:53:08.248219 22927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:53:08.251621 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 22927
I20260809 04:53:08.252383 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:34783
--local_ip_for_outbound_sockets=127.31.93.3
--tserver_master_addrs=127.31.93.62:44725
--webserver_port=34873
--webserver_interface=127.31.93.3
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:53:08 dist-test-slave-4tdj krb5kdc[21824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251188, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:53:08.272835 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:08.250983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:56587 (local address 127.31.93.62:44725)
0809 04:53:08.251214 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:08.251221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:08.252611 (+ 1390us) server_negotiation.cc:407] Connection header received
0809 04:53:08.255283 (+ 2672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:08.255289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:08.255374 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:08.255478 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:08.257407 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.258424 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.260744 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.261352 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.261468 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:08.265746 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:08.265754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:08.265759 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:08.265803 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:08.269331 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:08.270196 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:08.270202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:08.270206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:08.270407 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:08.270860 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:08.270865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:08.270869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:08.271248 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:08.271392 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:08.272196 (+ 804us) server_negotiation.cc:299] Negotiation successful
0809 04:53:08.272441 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 04:53:08.273897 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:08.251519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:56587)
0809 04:53:08.252441 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:08.252498 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:08.254957 (+ 2459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:08.255730 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:08.255742 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:08.256456 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:53:08.257208 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:08.257232 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.258592 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.258598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:08.260579 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:08.260595 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.261579 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.261586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:08.261866 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:08.262635 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:53:08.262664 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:53:08.265519 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:53:08.269487 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:08.269497 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:08.269512 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:08.270088 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:08.270525 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:08.270531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:08.270534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:08.270759 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:08.271405 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:53:08.271412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:53:08.271783 (+ 371us) client_negotiation.cc:769] Sending connection context
0809 04:53:08.272063 (+ 280us) client_negotiation.cc:240] Negotiation successful
0809 04:53:08.272587 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":375,"threads_started":1}
I20260809 04:53:08.275529 23109 heartbeater.cc:347] Connected to a master server at 127.31.93.62:44725
I20260809 04:53:08.275871 23109 heartbeater.cc:464] Registering TS with master...
I20260809 04:53:08.276806 23109 heartbeater.cc:511] Master 127.31.93.62:44725 requested a full tablet report, sending...
I20260809 04:53:08.278509 22642 ts_manager.cc:194] Registered new tserver with Master: a84edde35edf48e9ad12018132c550b4 (127.31.93.2:46529)
I20260809 04:53:08.279652 22642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:56587
W20260809 04:53:08.577510 23113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:08.577786 23113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:08.577919 23113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:08.584877 23113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:08.584990 23113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:08.585134 23113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:53:08.594507 23113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:34783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=34873
--tserver_master_addrs=127.31.93.62:44725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:08.595727 23113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:08.596974 23113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:08.609463 23121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:08.609441 23123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:08.610054 23126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:08.610112 23113 server_base.cc:1034] running on GCE node
I20260809 04:53:08.610746 23113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:08.611557 23113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:08.612756 23113 hybrid_clock.cc:648] HybridClock initialized: now 1786251188612718 us; error 30 us; skew 500 ppm
I20260809 04:53:08.613011 23113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:08 dist-test-slave-4tdj krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251188, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:08.615988 23113 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:53:08.617261 23113 webserver.cc:460] Webserver started at http://127.31.93.3:34873/ using document root <none> and password file <none>
I20260809 04:53:08.617882 23113 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:08.618007 23113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:08.623241 23113 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 04:53:08.625751 23134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:08.627094 23113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:53:08.627251 23113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "a8ff1545118c4085883aedeec0da494d"
format_stamp: "Formatted at 2026-08-09 04:53:05 on dist-test-slave-4tdj"
server_key: "ae43678b8ad672020036812069fe655e"
server_key_iv: "53feb349480fd92f9af566964c586098"
server_key_version: "encryptionkey@0"
I20260809 04:53:08.628178 23113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:08.646512 23113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:08.650382 23113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:08.650629 23113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:08.651497 23113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:53:08.652722 23113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:53:08.652828 23113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:08.652931 23113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:53:08.653007 23113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:08.682628 23113 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:34783
I20260809 04:53:08.682699 23300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:34783 every 8 connection(s)
I20260809 04:53:08.684029 23113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:53:08.689257 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 23113
Aug 09 04:53:08 dist-test-slave-4tdj krb5kdc[21824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251188, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:53:08.708875 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:08.690157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.31.93.62:44725)
0809 04:53:08.694128 (+ 3971us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:08.694134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:08.694516 (+ 382us) server_negotiation.cc:407] Connection header received
0809 04:53:08.694697 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:08.694700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:08.694753 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:08.694832 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:08.696617 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.697556 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.699355 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.700105 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.700192 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:08.703259 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:08.703265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:08.703270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:08.703310 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:08.706170 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:08.706978 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:08.706982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:08.706985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:08.707146 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:08.707538 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:08.707543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:08.707546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:08.707860 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:08.708106 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:08.708379 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 04:53:08.708538 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":3850,"thread_start_us":3742,"threads_started":1}
I20260809 04:53:08.710191 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:08.687101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:48045 (local address 127.31.93.62:44725)
0809 04:53:08.687282 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:08.687288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:08.689918 (+ 2630us) server_negotiation.cc:407] Connection header received
0809 04:53:08.693939 (+ 4021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:08.693946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:08.694047 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:08.694150 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:08.696203 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.697161 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.699275 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.700009 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.700132 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:08.703994 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:08.704000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:08.704004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:08.704043 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:08.706960 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:08.707707 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:08.707711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:08.707713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:08.707838 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:08.708324 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:08.708329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:08.708332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:08.708676 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:08.708816 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:08.709661 (+ 845us) server_negotiation.cc:299] Negotiation successful
0809 04:53:08.709865 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":63,"spinlock_wait_cycles":2304}
I20260809 04:53:08.711275 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:08.687857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:48045)
0809 04:53:08.688787 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:08.688837 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:08.693423 (+ 4586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:08.694496 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:08.694507 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:08.695234 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:53:08.696015 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:08.696041 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.697321 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.697325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:08.699139 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:08.699155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.700328 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.700338 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:08.700672 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:08.701467 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:53:08.701495 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:53:08.703745 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:53:08.707087 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:08.707096 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:08.707111 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:08.707588 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:08.707944 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:08.707949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:08.707951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:08.708200 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:08.708833 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:53:08.708841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:53:08.709240 (+ 399us) client_negotiation.cc:769] Sending connection context
0809 04:53:08.709526 (+ 286us) client_negotiation.cc:240] Negotiation successful
0809 04:53:08.709951 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":363,"threads_started":1}
I20260809 04:53:08.712630 23301 heartbeater.cc:347] Connected to a master server at 127.31.93.62:44725
I20260809 04:53:08.712883 23301 heartbeater.cc:464] Registering TS with master...
I20260809 04:53:08.713538 23301 heartbeater.cc:511] Master 127.31.93.62:44725 requested a full tablet report, sending...
I20260809 04:53:08.714814 22642 ts_manager.cc:194] Registered new tserver with Master: a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783)
I20260809 04:53:08.715590 22642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:48045
I20260809 04:53:08.723933 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:53:08.730104 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:08.724849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.31.93.62:44725)
0809 04:53:08.725052 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:08.725058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:08.725337 (+ 279us) server_negotiation.cc:407] Connection header received
0809 04:53:08.725513 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:08.725517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:08.725663 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:08.725777 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:53:08.726122 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.726913 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.728247 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.728629 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.728714 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:08.729177 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:53:08.729643 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:53:08.729744 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:08.729798 (+ 54us) server_negotiation.cc:299] Negotiation successful
0809 04:53:08.729856 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 04:53:08.730990 22642 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:53:08.738010 22642 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:53:08.760475 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:08.748942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:39573 (local address 127.0.0.1:46408)
0809 04:53:08.750686 (+ 1744us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:08.750738 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:08.750923 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:08.752022 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:08.752031 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:08.752064 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:08.752488 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:08.752501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.754610 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.754616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:08.757152 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:08.757170 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.759431 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.759438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:08.759680 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:08.759732 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 04:53:08.759868 (+ 136us) client_negotiation.cc:240] Negotiation successful
0809 04:53:08.760057 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1474,"thread_start_us":1300,"threads_started":1}
I20260809 04:53:08.761564 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:08.749038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.31.93.1:39573)
0809 04:53:08.750195 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:08.750206 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:08.751119 (+ 913us) server_negotiation.cc:407] Connection header received
0809 04:53:08.751236 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:08.751245 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:08.751514 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:08.751718 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:08.752721 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.754417 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.757353 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.758933 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.759060 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:08.759271 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:08.761048 (+ 1777us) server_negotiation.cc:299] Negotiation successful
0809 04:53:08.761237 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":984,"thread_start_us":1232,"threads_started":1}
I20260809 04:53:08.766783 22817 tablet_service.cc:1467] Processing CreateTablet for tablet 96f2af6e5f37414298e09b8545f37e46 (DEFAULT_TABLE table=test-table [id=ede6ef814b2f4dd49ee2b46b80f0fb5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:53:08.768348 22817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96f2af6e5f37414298e09b8545f37e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:53:08.769205 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:08.749876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:34783 (local address 127.0.0.1:49330)
0809 04:53:08.760741 (+ 10865us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:08.760762 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:08.760870 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:08.762597 (+ 1727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:08.762602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:08.762627 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:08.762792 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:08.762803 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.764685 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.764690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:08.766890 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:08.766906 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.768457 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.768463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:08.768648 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:08.768680 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 04:53:08.768764 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 04:53:08.768894 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10747,"mutex_wait_us":2,"thread_start_us":12931,"threads_started":1}
I20260809 04:53:08.769647 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:08.750007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.31.93.3:34783)
0809 04:53:08.761954 (+ 11947us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:08.761963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:08.761997 (+ 34us) server_negotiation.cc:407] Connection header received
0809 04:53:08.762095 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:08.762103 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:08.762369 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:08.762589 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:08.762976 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.764526 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.767066 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.768293 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.768434 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:08.768661 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:08.768805 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 04:53:08.768997 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11755,"thread_start_us":11618,"threads_started":1}
I20260809 04:53:08.770568 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:08.751678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:46529 (local address 127.0.0.1:38142)
0809 04:53:08.762172 (+ 10494us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:08.762190 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:08.762324 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:08.763196 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:08.763201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:08.763228 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:08.763550 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:08.763562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.765306 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.765311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:08.767319 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:08.767334 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.769557 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.769564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:08.769726 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:08.769757 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 04:53:08.769848 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 04:53:08.769967 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10359,"mutex_wait_us":1,"thread_start_us":10391,"threads_started":1}
I20260809 04:53:08.771281 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:08.751401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.31.93.2:46529)
0809 04:53:08.762606 (+ 11205us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:08.762614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:08.762645 (+ 31us) server_negotiation.cc:407] Connection header received
0809 04:53:08.762734 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:08.762741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:08.762967 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:08.763730 (+ 763us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:08.764001 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.765162 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.767497 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:08.768937 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:08.769150 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:08.769362 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:08.770787 (+ 1425us) server_negotiation.cc:299] Negotiation successful
0809 04:53:08.770950 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11034,"thread_start_us":11003,"threads_started":1}
I20260809 04:53:08.773331 23206 tablet_service.cc:1467] Processing CreateTablet for tablet 96f2af6e5f37414298e09b8545f37e46 (DEFAULT_TABLE table=test-table [id=ede6ef814b2f4dd49ee2b46b80f0fb5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:53:08.774489 23206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96f2af6e5f37414298e09b8545f37e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:53:08.774914 23021 tablet_service.cc:1467] Processing CreateTablet for tablet 96f2af6e5f37414298e09b8545f37e46 (DEFAULT_TABLE table=test-table [id=ede6ef814b2f4dd49ee2b46b80f0fb5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:53:08.776578 23021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96f2af6e5f37414298e09b8545f37e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:53:08.786866 23322 tablet_bootstrap.cc:492] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673: Bootstrap starting.
I20260809 04:53:08.790961 23322 tablet_bootstrap.cc:654] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673: Neither blocks nor log segments found. Creating new log.
I20260809 04:53:08.791965 23323 tablet_bootstrap.cc:492] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d: Bootstrap starting.
I20260809 04:53:08.792969 23322 log.cc:824] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:08.795771 23322 tablet_bootstrap.cc:492] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673: No bootstrap required, opened a new log
I20260809 04:53:08.796025 23322 ts_tablet_manager.cc:1397] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260809 04:53:08.796458 23323 tablet_bootstrap.cc:654] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d: Neither blocks nor log segments found. Creating new log.
I20260809 04:53:08.797586 23323 log.cc:824] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:08.800518 23323 tablet_bootstrap.cc:492] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d: No bootstrap required, opened a new log
I20260809 04:53:08.800784 23323 ts_tablet_manager.cc:1397] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260809 04:53:08.802359 23324 tablet_bootstrap.cc:492] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4: Bootstrap starting.
I20260809 04:53:08.805598 23324 tablet_bootstrap.cc:654] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4: Neither blocks nor log segments found. Creating new log.
I20260809 04:53:08.806142 23322 raft_consensus.cc:348] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } }
I20260809 04:53:08.806603 23322 raft_consensus.cc:374] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:53:08.806715 23322 raft_consensus.cc:729] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c564c786e65e40cb8b28b23f94102673, State: Initialized, Role: FOLLOWER
I20260809 04:53:08.806864 23324 log.cc:824] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:08.807438 23322 consensus_queue.cc:260] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } }
I20260809 04:53:08.808459 22920 heartbeater.cc:503] Master 127.31.93.62:44725 was elected leader, sending a full tablet report...
I20260809 04:53:08.808923 23322 ts_tablet_manager.cc:1428] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260809 04:53:08.809773 23324 tablet_bootstrap.cc:492] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4: No bootstrap required, opened a new log
I20260809 04:53:08.809700 23323 raft_consensus.cc:348] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } }
I20260809 04:53:08.809998 23324 ts_tablet_manager.cc:1397] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260809 04:53:08.810110 23323 raft_consensus.cc:374] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:53:08.810227 23323 raft_consensus.cc:729] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ff1545118c4085883aedeec0da494d, State: Initialized, Role: FOLLOWER
I20260809 04:53:08.810855 23323 consensus_queue.cc:260] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } }
I20260809 04:53:08.813197 23323 ts_tablet_manager.cc:1428] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260809 04:53:08.813526 23301 heartbeater.cc:503] Master 127.31.93.62:44725 was elected leader, sending a full tablet report...
I20260809 04:53:08.817847 23324 raft_consensus.cc:348] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } }
I20260809 04:53:08.818145 23324 raft_consensus.cc:374] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:53:08.818236 23324 raft_consensus.cc:729] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84edde35edf48e9ad12018132c550b4, State: Initialized, Role: FOLLOWER
I20260809 04:53:08.818750 23324 consensus_queue.cc:260] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } }
I20260809 04:53:08.819648 23109 heartbeater.cc:503] Master 127.31.93.62:44725 was elected leader, sending a full tablet report...
I20260809 04:53:08.820034 23324 ts_tablet_manager.cc:1428] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
W20260809 04:53:08.941277 23303 tablet.cc:2369] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:53:09.000080 23110 tablet.cc:2369] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:53:09.003993 23332 raft_consensus.cc:482] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:53:09.004264 23332 raft_consensus.cc:504] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } }
I20260809 04:53:09.005911 23332 leader_election.cc:288] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a84edde35edf48e9ad12018132c550b4 (127.31.93.2:46529), a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783)
I20260809 04:53:09.012370 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:09.006330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:36557 (local address 127.31.93.2:46529)
0809 04:53:09.006565 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:09.006571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:09.007250 (+ 679us) server_negotiation.cc:407] Connection header received
0809 04:53:09.007414 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:09.007418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:09.007490 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:09.007581 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:09.008139 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:09.008809 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:09.010730 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:09.011490 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:09.011596 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:09.011697 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:09.011996 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 04:53:09.012129 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 04:53:09.012388 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:09.006553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:37339 (local address 127.31.93.3:34783)
0809 04:53:09.006739 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:09.006744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:09.007038 (+ 294us) server_negotiation.cc:407] Connection header received
0809 04:53:09.007243 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:09.007249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:09.007322 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:09.007405 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:09.008048 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:09.008791 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:09.010752 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:09.011505 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:09.011623 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:09.011719 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:09.012050 (+ 331us) server_negotiation.cc:299] Negotiation successful
0809 04:53:09.012166 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 04:53:09.012495 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:09.006285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:46529 (local address 127.31.93.1:36557)
0809 04:53:09.007139 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:09.007152 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:09.007281 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:09.007609 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:09.007612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:09.007646 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:09.007993 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:09.008004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:09.008941 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:09.008945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:09.010596 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:09.010608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:09.011707 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:09.011713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:09.011855 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:09.011878 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:53:09.011949 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 04:53:09.012061 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":16,"thread_start_us":555,"threads_started":1}
I20260809 04:53:09.012558 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:09.006179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:34783 (local address 127.31.93.1:37339)
0809 04:53:09.006906 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:09.006937 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:09.007106 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:09.007445 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:09.007450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:09.007473 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:09.007889 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:09.007902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:09.008918 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:09.008923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:09.010611 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:09.010623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:09.011755 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:09.011760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:09.011894 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:09.011916 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 04:53:09.012005 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 04:53:09.012128 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":70,"spinlock_wait_cycles":1920,"thread_start_us":391,"threads_started":1}
I20260809 04:53:09.013561 23045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f2af6e5f37414298e09b8545f37e46" candidate_uuid: "c564c786e65e40cb8b28b23f94102673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a84edde35edf48e9ad12018132c550b4" is_pre_election: true
I20260809 04:53:09.013671 23234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f2af6e5f37414298e09b8545f37e46" candidate_uuid: "c564c786e65e40cb8b28b23f94102673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ff1545118c4085883aedeec0da494d" is_pre_election: true
I20260809 04:53:09.014144 23045 raft_consensus.cc:2454] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c564c786e65e40cb8b28b23f94102673 in term 0.
I20260809 04:53:09.014184 23234 raft_consensus.cc:2454] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c564c786e65e40cb8b28b23f94102673 in term 0.
I20260809 04:53:09.014874 22755 leader_election.cc:302] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a84edde35edf48e9ad12018132c550b4, c564c786e65e40cb8b28b23f94102673; no voters:
I20260809 04:53:09.015255 23332 raft_consensus.cc:2781] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:53:09.015384 23332 raft_consensus.cc:482] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:53:09.015473 23332 raft_consensus.cc:3037] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:53:09.017551 23332 raft_consensus.cc:504] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } }
I20260809 04:53:09.018081 23332 leader_election.cc:288] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 1 election: Requested vote from peers a84edde35edf48e9ad12018132c550b4 (127.31.93.2:46529), a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783)
I20260809 04:53:09.018501 23045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f2af6e5f37414298e09b8545f37e46" candidate_uuid: "c564c786e65e40cb8b28b23f94102673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a84edde35edf48e9ad12018132c550b4"
I20260809 04:53:09.018510 23234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f2af6e5f37414298e09b8545f37e46" candidate_uuid: "c564c786e65e40cb8b28b23f94102673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ff1545118c4085883aedeec0da494d"
I20260809 04:53:09.018666 23234 raft_consensus.cc:3037] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:53:09.018662 23045 raft_consensus.cc:3037] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:53:09.021410 23234 raft_consensus.cc:2454] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c564c786e65e40cb8b28b23f94102673 in term 1.
I20260809 04:53:09.021781 23045 raft_consensus.cc:2454] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c564c786e65e40cb8b28b23f94102673 in term 1.
I20260809 04:53:09.021833 22753 leader_election.cc:302] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8ff1545118c4085883aedeec0da494d, c564c786e65e40cb8b28b23f94102673; no voters:
I20260809 04:53:09.022194 23332 raft_consensus.cc:2781] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:53:09.031574 23332 raft_consensus.cc:686] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 LEADER]: Becoming Leader. State: Replica: c564c786e65e40cb8b28b23f94102673, State: Running, Role: LEADER
I20260809 04:53:09.032225 23332 consensus_queue.cc:237] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } }
I20260809 04:53:09.037981 22642 catalog_manager.cc:5495] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 reported cstate change: term changed from 0 to 1, leader changed from <none> to c564c786e65e40cb8b28b23f94102673 (127.31.93.1). New cstate: current_term: 1 leader_uuid: "c564c786e65e40cb8b28b23f94102673" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
W20260809 04:53:09.042960 22922 tablet.cc:2369] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:53:09.065968 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:09.060911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.31.93.1:39573)
0809 04:53:09.061099 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:09.061104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:09.061122 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:53:09.061302 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:09.061307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:09.061390 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:09.061523 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:53:09.061869 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:09.062621 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:09.063963 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:09.064361 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:09.064465 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:09.064925 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:53:09.065411 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:53:09.065547 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:09.065642 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 04:53:09.065718 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:53:09.073838 23234 raft_consensus.cc:1261] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d [term 1 FOLLOWER]: Refusing update from remote peer c564c786e65e40cb8b28b23f94102673: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:53:09.073838 23045 raft_consensus.cc:1261] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4 [term 1 FOLLOWER]: Refusing update from remote peer c564c786e65e40cb8b28b23f94102673: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:53:09.074898 23332 consensus_queue.cc:1035] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:53:09.075182 23338 consensus_queue.cc:1035] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:53:09.105520 22640 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:53:09.105774 22640 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:53:09.109169 22640 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c: Sending DeleteTablet for 3 replicas of tablet 96f2af6e5f37414298e09b8545f37e46
I20260809 04:53:09.110333 23206 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f2af6e5f37414298e09b8545f37e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:53:09 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:49330
I20260809 04:53:09.110366 23021 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f2af6e5f37414298e09b8545f37e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:53:09 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:38142
I20260809 04:53:09.110574 22817 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f2af6e5f37414298e09b8545f37e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:53:09 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:46408
I20260809 04:53:09.111294 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 22701
I20260809 04:53:09.122073 23357 tablet_replica.cc:330] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d: stopping tablet replica
I20260809 04:53:09.122658 23357 raft_consensus.cc:2229] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:53:09.123261 23357 raft_consensus.cc:2258] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:53:09.124125 23361 tablet_replica.cc:330] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673: stopping tablet replica
I20260809 04:53:09.124533 23361 raft_consensus.cc:2229] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:53:09.125180 23361 raft_consensus.cc:2258] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:53:09.127316 23357 ts_tablet_manager.cc:1905] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:53:09.129808 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:09.112340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46420 (local address 127.31.93.1:39573)
0809 04:53:09.112512 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:09.112519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:09.112538 (+ 19us) server_negotiation.cc:407] Connection header received
0809 04:53:09.112594 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:09.112599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:09.112678 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:09.112778 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:09.114508 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:09.115830 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:09.117731 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:09.118445 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:09.118568 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:09.121861 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:09.121874 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:09.121892 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:09.121937 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:09.126675 (+ 4738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:09.127520 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:09.127528 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:09.127532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:09.127687 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:09.128079 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:09.128084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:09.128087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:09.128484 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:09.128806 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:09.129085 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 04:53:09.129305 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:53:09.130723 22913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:09.131560 23355 tablet_replica.cc:330] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4: stopping tablet replica
I20260809 04:53:09.131963 23355 raft_consensus.cc:2229] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:53:09.132440 23355 raft_consensus.cc:2258] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:53:09.134320 23357 ts_tablet_manager.cc:1918] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:53:09.134531 23357 log.cc:1196] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/96f2af6e5f37414298e09b8545f37e46
I20260809 04:53:09.135084 23357 ts_tablet_manager.cc:1939] T 96f2af6e5f37414298e09b8545f37e46 P a8ff1545118c4085883aedeec0da494d: Deleting consensus metadata
I20260809 04:53:09.135560 23355 ts_tablet_manager.cc:1905] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:53:09.136653 22626 catalog_manager.cc:4841] TS a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783): tablet 96f2af6e5f37414298e09b8545f37e46 (table test-table [id=ede6ef814b2f4dd49ee2b46b80f0fb5d]) successfully deleted
I20260809 04:53:09.142309 23355 ts_tablet_manager.cc:1918] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:53:09.142460 23355 log.cc:1196] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/96f2af6e5f37414298e09b8545f37e46
I20260809 04:53:09.142952 23355 ts_tablet_manager.cc:1939] T 96f2af6e5f37414298e09b8545f37e46 P a84edde35edf48e9ad12018132c550b4: Deleting consensus metadata
I20260809 04:53:09.144537 22627 catalog_manager.cc:4841] TS a84edde35edf48e9ad12018132c550b4 (127.31.93.2:46529): tablet 96f2af6e5f37414298e09b8545f37e46 (table test-table [id=ede6ef814b2f4dd49ee2b46b80f0fb5d]) successfully deleted
I20260809 04:53:09.226740 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 22701
W20260809 04:53:09.249681 22627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.93.1:39573 (error 108)
I20260809 04:53:09.250015 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 22927
I20260809 04:53:09.251678 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:09.251178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:39573 (local address 127.0.0.1:46424)
0809 04:53:09.251394 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:09.251512 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260809 04:53:09.251940 22627 catalog_manager.cc:4584] TS c564c786e65e40cb8b28b23f94102673 (127.31.93.1:39573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96f2af6e5f37414298e09b8545f37e46: Network error: Client connection negotiation failed: client connection to 127.31.93.1:39573: connect: Connection refused (error 111)
I20260809 04:53:09.263876 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:09.250773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.31.93.2:46529)
0809 04:53:09.250935 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:09.250940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:09.251093 (+ 153us) server_negotiation.cc:407] Connection header received
0809 04:53:09.251277 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:09.251282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:09.251384 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:09.251495 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:09.252998 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:09.253726 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:09.255190 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:09.255637 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:09.255753 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:09.258193 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:09.258204 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:09.258216 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:09.258247 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:09.261083 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:09.261726 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:09.261732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:09.261735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:09.261856 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:09.262278 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:09.262283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:09.262287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:09.262731 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:09.263006 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:09.263332 (+ 326us) server_negotiation.cc:299] Negotiation successful
0809 04:53:09.263546 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:53:09.264467 23103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:09.305615 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:09.305093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:39573 (local address 127.0.0.1:46440)
0809 04:53:09.305314 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:09.305427 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 04:53:09.306696 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:09.306316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:39573 (local address 127.0.0.1:46452)
0809 04:53:09.306457 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:09.306546 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260809 04:53:09.351112 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 22927
I20260809 04:53:09.367049 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:09.366579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:39573 (local address 127.0.0.1:46456)
0809 04:53:09.366800 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:09.366884 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 04:53:09.368060 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:09.367701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:39573 (local address 127.0.0.1:46460)
0809 04:53:09.367846 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:09.367928 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260809 04:53:09.380059 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 23113
I20260809 04:53:09.396888 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:09.384741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.31.93.3:34783)
0809 04:53:09.384917 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:09.384923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:09.384944 (+ 21us) server_negotiation.cc:407] Connection header received
0809 04:53:09.385093 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:09.385097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:09.385162 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:09.385270 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:09.386712 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:09.387406 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:09.388746 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:09.389137 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:09.389243 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:09.391734 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:09.391750 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:09.391768 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:09.391802 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:09.394249 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:09.394956 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:09.394964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:09.394966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:09.395112 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:09.395471 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:09.395475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:09.395477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:09.395798 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:09.396087 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:09.396376 (+ 289us) server_negotiation.cc:299] Negotiation successful
0809 04:53:09.396588 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:53:09.397504 23289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:09.449146 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:09.448641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:39573 (local address 127.0.0.1:46472)
0809 04:53:09.448876 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:09.448977 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260809 04:53:09.450192 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:09.449823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:39573 (local address 127.0.0.1:46476)
0809 04:53:09.450009 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:09.450086 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260809 04:53:09.485862 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 23113
I20260809 04:53:09.507829 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 22600
I20260809 04:53:09.508967 22688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:09.618361 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 22600
I20260809 04:53:09.642915 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:44725
--webserver_interface=127.31.93.62
--webserver_port=33647
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:44725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:53:09.969317 23368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:09.969615 23368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:09.969769 23368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:09.977044 23368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:53:09.977164 23368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:09.977252 23368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:09.977344 23368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:53:09.977409 23368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:53:09.987187 23368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:44725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:44725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=33647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:09.988458 23368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:09.989807 23368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:10.001878 23382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:10.002010 23376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:10.002084 23375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:10.003038 23368 server_base.cc:1034] running on GCE node
I20260809 04:53:10.003829 23368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:10.005366 23368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:10.006634 23368 hybrid_clock.cc:648] HybridClock initialized: now 1786251190006582 us; error 59 us; skew 500 ppm
I20260809 04:53:10.007025 23368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:10 dist-test-slave-4tdj krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251190, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:10.010147 23368 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:53:10.011695 23368 webserver.cc:460] Webserver started at http://127.31.93.62:33647/ using document root <none> and password file <none>
I20260809 04:53:10.012373 23368 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:10.012491 23368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:10.017838 23368 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:53:10.020674 23398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:10.021894 23368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:53:10.022051 23368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c"
format_stamp: "Formatted at 2026-08-09 04:53:04 on dist-test-slave-4tdj"
server_key: "57ea955e214b39468e3982c4fdca4242"
server_key_iv: "2606f6902ae7924d4097eca2d8d50fc2"
server_key_version: "encryptionkey@0"
I20260809 04:53:10.023018 23368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:10.035339 23368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:10.039243 23368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:10.039476 23368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:10.056471 23368 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:44725
I20260809 04:53:10.056489 23479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:44725 every 8 connection(s)
I20260809 04:53:10.057891 23368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:53:10.064545 23480 sys_catalog.cc:263] Verifying existing consensus state
I20260809 04:53:10.065452 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 23368
I20260809 04:53:10.066479 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:39573
--local_ip_for_outbound_sockets=127.31.93.1
--tserver_master_addrs=127.31.93.62:44725
--webserver_port=33539
--webserver_interface=127.31.93.1
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:53:10.068928 23480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c: Bootstrap starting.
I20260809 04:53:10.090744 23480 log.cc:824] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:10.106829 23480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:53:10.107313 23480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c: Bootstrap complete.
I20260809 04:53:10.115927 23480 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } }
I20260809 04:53:10.116771 23480 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd94ed2317924c2c9c1dfd6caf15aa3c, State: Initialized, Role: FOLLOWER
I20260809 04:53:10.117383 23480 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } }
I20260809 04:53:10.117592 23480 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:53:10.117705 23480 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:53:10.117863 23480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 2 FOLLOWER]: Advancing to term 3
I20260809 04:53:10.121428 23480 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } }
I20260809 04:53:10.121863 23480 leader_election.cc:302] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd94ed2317924c2c9c1dfd6caf15aa3c; no voters:
I20260809 04:53:10.122471 23480 leader_election.cc:288] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 04:53:10.122651 23488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 04:53:10.123706 23488 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [term 3 LEADER]: Becoming Leader. State: Replica: fd94ed2317924c2c9c1dfd6caf15aa3c, State: Running, Role: LEADER
I20260809 04:53:10.124260 23488 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } }
I20260809 04:53:10.124571 23480 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:53:10.126652 23491 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd94ed2317924c2c9c1dfd6caf15aa3c. Latest consensus state: current_term: 3 leader_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } } }
I20260809 04:53:10.127020 23491 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [sys.catalog]: This master's current role is: LEADER
I20260809 04:53:10.127614 23489 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd94ed2317924c2c9c1dfd6caf15aa3c" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44725 } } }
I20260809 04:53:10.127867 23489 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd94ed2317924c2c9c1dfd6caf15aa3c [sys.catalog]: This master's current role is: LEADER
I20260809 04:53:10.135389 23496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:53:10.147595 23496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:53:10.149055 23496 catalog_manager.cc:1259] Loaded cluster ID: 3bbd1aa07c9a496a83c2d5d472f47132
I20260809 04:53:10.149192 23496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260809 04:53:10.151249 23496 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa36fbe39ad google::LogMessage::Fail() at ??:0
@ 0x7fa36fbe688a google::LogMessage::SendToLog() at ??:0
@ 0x7fa36fbe34de google::LogMessage::Flush() at ??:0
@ 0x7fa36fbe4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa385616512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa385613976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa38569b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa38569b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa37282f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa37283291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa372832799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa37280deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa375ae76db start_thread at ??:0
@ 0x7fa36ea7d61f clone at ??:0
W20260809 04:53:10.413046 23484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:10.413311 23484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:10.413442 23484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:10.420430 23484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:10.420543 23484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:10.420703 23484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:53:10.430838 23484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:39573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=33539
--tserver_master_addrs=127.31.93.62:44725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:10.432014 23484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:10.433327 23484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:10.445497 23544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:10.445437 23545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:10.445411 23548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:10.446455 23484 server_base.cc:1034] running on GCE node
I20260809 04:53:10.447029 23484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:10.447865 23484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:10.449057 23484 hybrid_clock.cc:648] HybridClock initialized: now 1786251190449004 us; error 46 us; skew 500 ppm
I20260809 04:53:10.449350 23484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:10 dist-test-slave-4tdj krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251190, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:10.452226 23484 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:53:10.453560 23484 webserver.cc:460] Webserver started at http://127.31.93.1:33539/ using document root <none> and password file <none>
I20260809 04:53:10.454195 23484 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:10.454329 23484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:10.459476 23484 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260809 04:53:10.462348 23556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:10.463688 23484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:53:10.463838 23484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "c564c786e65e40cb8b28b23f94102673"
format_stamp: "Formatted at 2026-08-09 04:53:04 on dist-test-slave-4tdj"
server_key: "82176f520017a9148bc2b32ad5abf7f1"
server_key_iv: "6a1c013c7c7bf38393d36ebf3c97e826"
server_key_version: "encryptionkey@0"
I20260809 04:53:10.464764 23484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:10.481554 23484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:10.485494 23484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:10.485738 23484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:10.486709 23484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:53:10.488384 23569 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260809 04:53:10.493829 23484 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 04:53:10.493950 23484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260809 04:53:10.494058 23484 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 04:53:10.497263 23484 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 04:53:10.497341 23484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260809 04:53:10.497627 23569 tablet_bootstrap.cc:492] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673: Bootstrap starting.
I20260809 04:53:10.505324 23569 log.cc:824] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:10.518942 23569 tablet_bootstrap.cc:492] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:53:10.519544 23569 tablet_bootstrap.cc:492] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673: Bootstrap complete.
I20260809 04:53:10.519939 23569 ts_tablet_manager.cc:1397] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673: Time spent bootstrapping tablet: real 0.023s user 0.021s sys 0.000s
I20260809 04:53:10.527153 23569 raft_consensus.cc:348] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } }
I20260809 04:53:10.528507 23569 raft_consensus.cc:729] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c564c786e65e40cb8b28b23f94102673, State: Initialized, Role: FOLLOWER
I20260809 04:53:10.529131 23569 consensus_queue.cc:260] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } }
I20260809 04:53:10.530584 23569 ts_tablet_manager.cc:1428] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673: Time spent starting tablet: real 0.011s user 0.002s sys 0.009s
I20260809 04:53:10.539191 23484 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:39573
I20260809 04:53:10.539252 23731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:39573 every 8 connection(s)
I20260809 04:53:10.540771 23484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:53:10.544862 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:10.543177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:37335)
0809 04:53:10.543903 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:10.544112 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":305,"threads_started":1}
I20260809 04:53:10.545086 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 23484
W20260809 04:53:10.545456 23733 heartbeater.cc:644] Failed to heartbeat to 127.31.93.62:44725 (0 consecutive failures): Network error: Failed to ping master at 127.31.93.62:44725: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
I20260809 04:53:10.545759 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:46529
--local_ip_for_outbound_sockets=127.31.93.2
--tserver_master_addrs=127.31.93.62:44725
--webserver_port=32931
--webserver_interface=127.31.93.2
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:53:10.546243 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:10.545905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:43873)
0809 04:53:10.546081 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:10.546138 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260809 04:53:10.547168 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:10.546756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:43783)
0809 04:53:10.546922 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:10.547015 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 04:53:10.547435 23733 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 04:53:10.869333 23738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:10.869659 23738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:10.869822 23738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:10.876989 23738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:10.877115 23738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:10.877288 23738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:53:10.886847 23738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:46529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=32931
--tserver_master_addrs=127.31.93.62:44725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:10.888105 23738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:10.889472 23738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:10.902209 23745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:10.902343 23747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:10.902369 23749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:10.903286 23738 server_base.cc:1034] running on GCE node
I20260809 04:53:10.903872 23738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:10.904707 23738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:10.905934 23738 hybrid_clock.cc:648] HybridClock initialized: now 1786251190905874 us; error 44 us; skew 500 ppm
I20260809 04:53:10.906216 23738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:10 dist-test-slave-4tdj krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251190, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:10.909221 23738 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:53:10.910540 23738 webserver.cc:460] Webserver started at http://127.31.93.2:32931/ using document root <none> and password file <none>
I20260809 04:53:10.911145 23738 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:10.911245 23738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:10.916177 23738 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260809 04:53:10.918695 23759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:10.919934 23738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:53:10.920063 23738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "a84edde35edf48e9ad12018132c550b4"
format_stamp: "Formatted at 2026-08-09 04:53:05 on dist-test-slave-4tdj"
server_key: "48406ead1b40a66c3f1977775faec31b"
server_key_iv: "a9c81c73cfc86493d447e61fda55863b"
server_key_version: "encryptionkey@0"
I20260809 04:53:10.920958 23738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:10.937811 23738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:10.941231 23738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:10.941578 23738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:10.942479 23738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:53:10.943706 23738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:53:10.943792 23738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:10.943872 23738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:53:10.943925 23738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:10.973215 23738 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:46529
I20260809 04:53:10.973258 23924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:46529 every 8 connection(s)
I20260809 04:53:10.974622 23738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:53:10.978087 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 23738
I20260809 04:53:10.978781 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:34783
--local_ip_for_outbound_sockets=127.31.93.3
--tserver_master_addrs=127.31.93.62:44725
--webserver_port=34873
--webserver_interface=127.31.93.3
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:53:10.979161 23928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:10.977301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:58449)
0809 04:53:10.978003 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:10.978182 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":294,"threads_started":1}
W20260809 04:53:10.979749 23925 heartbeater.cc:644] Failed to heartbeat to 127.31.93.62:44725 (0 consecutive failures): Network error: Failed to ping master at 127.31.93.62:44725: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
I20260809 04:53:10.980548 23928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:10.980253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:60505)
0809 04:53:10.980382 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:10.980448 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 04:53:10.981314 23928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:10.981024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:60623)
0809 04:53:10.981142 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:10.981215 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 04:53:10.981523 23925 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 04:53:11.292271 23931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:11.292515 23931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:11.292649 23931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:11.299594 23931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:11.299702 23931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:11.299850 23931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:53:11.308957 23931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:34783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=34873
--tserver_master_addrs=127.31.93.62:44725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:11.310071 23931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:11.311378 23931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:11.323300 23942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:11.323331 23940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:11.323266 23939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:11.324294 23931 server_base.cc:1034] running on GCE node
I20260809 04:53:11.324880 23931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:11.325754 23931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:11.326929 23931 hybrid_clock.cc:648] HybridClock initialized: now 1786251191326874 us; error 51 us; skew 500 ppm
I20260809 04:53:11.327198 23931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:11 dist-test-slave-4tdj krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251191, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:11.330037 23931 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:53:11.331302 23931 webserver.cc:460] Webserver started at http://127.31.93.3:34873/ using document root <none> and password file <none>
I20260809 04:53:11.331921 23931 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:11.332037 23931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:11.336884 23931 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260809 04:53:11.339406 23958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:11.340610 23931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 04:53:11.340755 23931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "a8ff1545118c4085883aedeec0da494d"
format_stamp: "Formatted at 2026-08-09 04:53:05 on dist-test-slave-4tdj"
server_key: "ae43678b8ad672020036812069fe655e"
server_key_iv: "53feb349480fd92f9af566964c586098"
server_key_version: "encryptionkey@0"
I20260809 04:53:11.341647 23931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:11.361182 23931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:11.364941 23931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:11.365171 23931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:11.366039 23931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:53:11.367268 23931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:53:11.367362 23931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:11.367461 23931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:53:11.367538 23931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:11.396886 23931 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:34783
I20260809 04:53:11.396965 24129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:34783 every 8 connection(s)
I20260809 04:53:11.398276 23931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:53:11.402720 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 23931
I20260809 04:53:11.403468 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:11.401391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:35105)
0809 04:53:11.402093 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:11.402319 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":269,"threads_started":1}
W20260809 04:53:11.404239 24130 heartbeater.cc:644] Failed to heartbeat to 127.31.93.62:44725 (0 consecutive failures): Network error: Failed to ping master at 127.31.93.62:44725: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
I20260809 04:53:11.405064 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:11.404768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:44093)
0809 04:53:11.404903 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:11.404965 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 04:53:11.405815 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:11.405537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:54187)
0809 04:53:11.405648 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:11.405716 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 04:53:11.406044 24130 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 04:53:11.548601 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:11.547887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:56189)
0809 04:53:11.548294 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:11.548403 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":184,"threads_started":1}
I20260809 04:53:11.753602 24138 raft_consensus.cc:482] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:53:11.754066 24138 raft_consensus.cc:504] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } }
I20260809 04:53:11.755760 24138 leader_election.cc:288] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a84edde35edf48e9ad12018132c550b4 (127.31.93.2:46529), a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783)
Aug 09 04:53:11 dist-test-slave-4tdj krb5kdc[21824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251190, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
Aug 09 04:53:11 dist-test-slave-4tdj krb5kdc[21824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251190, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:53:11.790748 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:11.756035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:34783 (local address 127.31.93.1:35311)
0809 04:53:11.771326 (+ 15291us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:11.771352 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:11.771477 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:11.775310 (+ 3833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:11.775315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:11.775855 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:53:11.776138 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:11.776150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:11.778412 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:11.778418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:11.779588 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:11.779600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:11.780525 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:11.780532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:11.780726 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:11.781244 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:53:11.781261 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:53:11.783476 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:53:11.787050 (+ 3574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:11.787059 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:11.787072 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:11.787469 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:11.787936 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:11.787940 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:11.787943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:11.788174 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:11.789441 (+ 1267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:53:11.789450 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:53:11.789601 (+ 151us) client_negotiation.cc:769] Sending connection context
0809 04:53:11.789832 (+ 231us) client_negotiation.cc:240] Negotiation successful
0809 04:53:11.790322 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15136,"mutex_wait_us":185,"spinlock_wait_cycles":16256,"thread_start_us":15838,"threads_started":1}
I20260809 04:53:11.791285 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:11.756463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:55885 (local address 127.31.93.2:46529)
0809 04:53:11.769885 (+ 13422us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:11.769904 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:11.769952 (+ 48us) server_negotiation.cc:407] Connection header received
0809 04:53:11.772233 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:11.772244 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:11.772642 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:11.772939 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:11.774917 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:11.776519 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:11.777978 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:11.778824 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:11.779013 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:11.783451 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:11.783462 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:11.783480 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:11.783526 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:11.787334 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:11.787977 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:11.787986 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:11.787993 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:11.788185 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:11.788609 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:11.788612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:11.788614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:11.789057 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:11.789220 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:11.790120 (+ 900us) server_negotiation.cc:299] Negotiation successful
0809 04:53:11.790541 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13215,"thread_start_us":15442,"threads_started":1}
I20260809 04:53:11.791378 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:11.756493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:35311 (local address 127.31.93.3:34783)
0809 04:53:11.772062 (+ 15569us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:11.772087 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:11.772129 (+ 42us) server_negotiation.cc:407] Connection header received
0809 04:53:11.774511 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:11.774524 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:11.774989 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:11.775335 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:11.776302 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:11.778238 (+ 1936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:11.779718 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:11.780327 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:11.780492 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:11.783713 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:11.783720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:11.783731 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:11.783765 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:11.786917 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:11.787589 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:11.787598 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:11.787606 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:11.787803 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:11.788626 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:11.788631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:11.788634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:11.789298 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:11.789474 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:11.790333 (+ 859us) server_negotiation.cc:299] Negotiation successful
0809 04:53:11.790704 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15385,"thread_start_us":15216,"threads_started":1}
I20260809 04:53:11.791427 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:11.756040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:46529 (local address 127.31.93.1:55885)
0809 04:53:11.761188 (+ 5148us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:11.761235 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:11.761816 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:11.773092 (+ 11276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:11.773102 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:11.773950 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:53:11.774736 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:11.774755 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:11.776681 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:11.776685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:11.777847 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:11.777859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:11.779043 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:11.779050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:11.779350 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:11.780089 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:53:11.780121 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:53:11.783140 (+ 3019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:53:11.787469 (+ 4329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:11.787475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:11.787478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:11.787833 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:11.788317 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:11.788322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:11.788326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:11.788491 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:11.789206 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:53:11.789213 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:53:11.789548 (+ 335us) client_negotiation.cc:769] Sending connection context
0809 04:53:11.789832 (+ 284us) client_negotiation.cc:240] Negotiation successful
0809 04:53:11.791107 (+ 1275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4984,"mutex_wait_us":887}
W20260809 04:53:11.794832 23565 leader_election.cc:341] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a84edde35edf48e9ad12018132c550b4 (127.31.93.2:46529): Not found: Tablet not found: 96f2af6e5f37414298e09b8545f37e46
W20260809 04:53:11.795149 23561 leader_election.cc:341] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783): Not found: Tablet not found: 96f2af6e5f37414298e09b8545f37e46
I20260809 04:53:11.795296 23561 leader_election.cc:302] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c564c786e65e40cb8b28b23f94102673; no voters: a84edde35edf48e9ad12018132c550b4, a8ff1545118c4085883aedeec0da494d
I20260809 04:53:11.795710 24138 raft_consensus.cc:2726] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:53:11.982873 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:11.981991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:54323)
0809 04:53:11.982506 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:11.982645 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":240,"threads_started":1}
I20260809 04:53:12.407193 24152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:12.406544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:58113)
0809 04:53:12.406954 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:12.407044 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":197,"threads_started":1}
I20260809 04:53:12.549983 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:12.549359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:55001)
0809 04:53:12.549762 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:12.549839 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":204,"threads_started":1}
I20260809 04:53:12.984356 24156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:12.983721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:45375)
0809 04:53:12.984130 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:12.984206 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":204,"threads_started":1}
I20260809 04:53:13.408666 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:13.407988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:57705)
0809 04:53:13.408415 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:13.408492 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":220,"threads_started":1}
I20260809 04:53:13.551445 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:13.550763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:55737)
0809 04:53:13.551192 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:13.551272 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":211,"threads_started":1}
I20260809 04:53:13.890202 24159 raft_consensus.cc:482] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:53:13.890446 24159 raft_consensus.cc:504] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } }
I20260809 04:53:13.891168 24159 leader_election.cc:288] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a84edde35edf48e9ad12018132c550b4 (127.31.93.2:46529), a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783)
W20260809 04:53:13.892221 23565 leader_election.cc:341] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a84edde35edf48e9ad12018132c550b4 (127.31.93.2:46529): Not found: Tablet not found: 96f2af6e5f37414298e09b8545f37e46
W20260809 04:53:13.892411 23561 leader_election.cc:341] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783): Not found: Tablet not found: 96f2af6e5f37414298e09b8545f37e46
I20260809 04:53:13.892513 23561 leader_election.cc:302] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c564c786e65e40cb8b28b23f94102673; no voters: a84edde35edf48e9ad12018132c550b4, a8ff1545118c4085883aedeec0da494d
I20260809 04:53:13.892746 24159 raft_consensus.cc:2726] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:53:13.985760 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:13.985118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:46357)
0809 04:53:13.985522 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:13.985607 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":209,"threads_started":1}
I20260809 04:53:14.410123 24162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:14.409408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:39935)
0809 04:53:14.409843 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:14.409923 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":227,"threads_started":1}
I20260809 04:53:14.552911 24163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:14.552207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:32781)
0809 04:53:14.552653 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:14.552730 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":241,"threads_started":1}
I20260809 04:53:14.987239 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:14.986594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:49357)
0809 04:53:14.986994 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:14.987070 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":191,"threads_started":1}
I20260809 04:53:15.411602 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:15.410919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:36223)
0809 04:53:15.411317 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:15.411395 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":198,"threads_started":1}
I20260809 04:53:15.554666 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:15.553869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:38915)
0809 04:53:15.554398 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:15.554490 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":248,"threads_started":1}
I20260809 04:53:15.988739 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:15.988057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:54047)
0809 04:53:15.988494 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:15.988572 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":219,"threads_started":1}
I20260809 04:53:16.413057 24170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:16.412384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:35033)
0809 04:53:16.412817 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:16.412906 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":236,"threads_started":1}
I20260809 04:53:16.556155 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:16.555474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:44447)
0809 04:53:16.555898 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:16.555979 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":208,"threads_started":1}
I20260809 04:53:16.585644 24173 raft_consensus.cc:482] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:53:16.585852 24173 raft_consensus.cc:504] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } }
I20260809 04:53:16.586524 24173 leader_election.cc:288] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a84edde35edf48e9ad12018132c550b4 (127.31.93.2:46529), a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783)
W20260809 04:53:16.587426 23565 leader_election.cc:341] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a84edde35edf48e9ad12018132c550b4 (127.31.93.2:46529): Not found: Tablet not found: 96f2af6e5f37414298e09b8545f37e46
W20260809 04:53:16.587606 23561 leader_election.cc:341] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783): Not found: Tablet not found: 96f2af6e5f37414298e09b8545f37e46
I20260809 04:53:16.587730 23561 leader_election.cc:302] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c564c786e65e40cb8b28b23f94102673; no voters: a84edde35edf48e9ad12018132c550b4, a8ff1545118c4085883aedeec0da494d
I20260809 04:53:16.587991 24173 raft_consensus.cc:2726] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:53:16.990270 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:16.989523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:57539)
0809 04:53:16.990016 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:16.990098 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":254,"threads_started":1}
I20260809 04:53:17.414592 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:17.413835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:58109)
0809 04:53:17.414286 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:17.414406 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":238,"threads_started":1}
I20260809 04:53:17.557567 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:17.556855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:58201)
0809 04:53:17.557296 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:17.557376 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":238,"threads_started":1}
I20260809 04:53:17.991834 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:17.991096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:47717)
0809 04:53:17.991573 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:17.991660 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":212,"threads_started":1}
I20260809 04:53:18.416144 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:18.415359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:44921)
0809 04:53:18.415852 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:18.415944 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":234,"threads_started":1}
I20260809 04:53:18.559024 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:18.558376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:50809)
0809 04:53:18.558787 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:18.558863 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":209,"threads_started":1}
I20260809 04:53:18.993369 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:18.992696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:53387)
0809 04:53:18.993108 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:18.993187 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":208,"threads_started":1}
I20260809 04:53:19.417568 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:19.416900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:57029)
0809 04:53:19.417330 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:19.417406 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":232,"threads_started":1}
I20260809 04:53:19.560436 24186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:19.559759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:50427)
0809 04:53:19.560194 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:19.560272 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":219,"threads_started":1}
I20260809 04:53:19.994921 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:19.994213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:39479)
0809 04:53:19.994693 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:19.994774 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":234,"threads_started":1}
I20260809 04:53:20.419056 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:20.418364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:40659)
0809 04:53:20.418816 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:20.418895 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":215,"threads_started":1}
I20260809 04:53:20.561916 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:20.561280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:32869)
0809 04:53:20.561690 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:20.561766 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":219,"threads_started":1}
I20260809 04:53:20.996408 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:20.995714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:46487)
0809 04:53:20.996153 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:20.996232 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":230,"threads_started":1}
I20260809 04:53:21.420497 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:21.419826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:57153)
0809 04:53:21.420253 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:21.420327 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":222,"threads_started":1}
I20260809 04:53:21.426491 24195 raft_consensus.cc:482] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:53:21.426673 24195 raft_consensus.cc:504] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } }
I20260809 04:53:21.427299 24195 leader_election.cc:288] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a84edde35edf48e9ad12018132c550b4 (127.31.93.2:46529), a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783)
W20260809 04:53:21.428081 23565 leader_election.cc:341] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a84edde35edf48e9ad12018132c550b4 (127.31.93.2:46529): Not found: Tablet not found: 96f2af6e5f37414298e09b8545f37e46
W20260809 04:53:21.428256 23561 leader_election.cc:341] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783): Not found: Tablet not found: 96f2af6e5f37414298e09b8545f37e46
I20260809 04:53:21.428351 23561 leader_election.cc:302] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c564c786e65e40cb8b28b23f94102673; no voters: a84edde35edf48e9ad12018132c550b4, a8ff1545118c4085883aedeec0da494d
I20260809 04:53:21.428557 24195 raft_consensus.cc:2726] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:53:21.563457 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:21.562702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:40037)
0809 04:53:21.563165 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:21.563262 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":214,"threads_started":1}
I20260809 04:53:21.997918 24198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:21.997221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:38955)
0809 04:53:21.997681 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:21.997767 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":216,"threads_started":1}
I20260809 04:53:22.421996 24199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:22.421290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:57119)
0809 04:53:22.421760 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:22.421839 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":244,"threads_started":1}
I20260809 04:53:22.564946 24200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:22.564251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:57553)
0809 04:53:22.564699 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:22.564780 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":226,"threads_started":1}
I20260809 04:53:22.999471 24202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:22.998763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:38155)
0809 04:53:22.999203 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:22.999282 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":216,"threads_started":1}
I20260809 04:53:23.423395 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:23.422754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:35039)
0809 04:53:23.423152 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:23.423228 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":207,"threads_started":1}
I20260809 04:53:23.566406 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:23.565690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:38559)
0809 04:53:23.566125 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:23.566199 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":210,"threads_started":1}
I20260809 04:53:24.001036 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:24.000267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:50983)
0809 04:53:24.000768 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:24.000864 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":259,"threads_started":1}
I20260809 04:53:24.424902 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:24.424203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:39983)
0809 04:53:24.424659 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:24.424747 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":238,"threads_started":1}
I20260809 04:53:24.567986 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:24.567222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:58867)
0809 04:53:24.567698 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:24.567785 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
I20260809 04:53:25.002652 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:25.001857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:51797)
0809 04:53:25.002339 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:25.002427 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":222,"threads_started":1}
I20260809 04:53:25.392581 24212 raft_consensus.cc:482] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:53:25.392773 24212 raft_consensus.cc:504] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c564c786e65e40cb8b28b23f94102673" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39573 } } peers { permanent_uuid: "a84edde35edf48e9ad12018132c550b4" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 46529 } } peers { permanent_uuid: "a8ff1545118c4085883aedeec0da494d" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34783 } }
I20260809 04:53:25.393424 24212 leader_election.cc:288] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a84edde35edf48e9ad12018132c550b4 (127.31.93.2:46529), a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783)
W20260809 04:53:25.394305 23565 leader_election.cc:341] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a84edde35edf48e9ad12018132c550b4 (127.31.93.2:46529): Not found: Tablet not found: 96f2af6e5f37414298e09b8545f37e46
W20260809 04:53:25.394481 23561 leader_election.cc:341] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8ff1545118c4085883aedeec0da494d (127.31.93.3:34783): Not found: Tablet not found: 96f2af6e5f37414298e09b8545f37e46
I20260809 04:53:25.394613 23561 leader_election.cc:302] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c564c786e65e40cb8b28b23f94102673; no voters: a84edde35edf48e9ad12018132c550b4, a8ff1545118c4085883aedeec0da494d
I20260809 04:53:25.394861 24212 raft_consensus.cc:2726] T 96f2af6e5f37414298e09b8545f37e46 P c564c786e65e40cb8b28b23f94102673 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:53:25.426401 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:25.425663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:58719)
0809 04:53:25.426099 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:25.426183 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":217,"threads_started":1}
I20260809 04:53:25.569814 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:25.569016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.1:37183)
0809 04:53:25.569510 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:25.569629 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":234,"threads_started":1}
I20260809 04:53:26.004303 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:26.003596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.2:37033)
0809 04:53:26.004038 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:26.004152 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":219,"threads_started":1}
I20260809 04:53:26.409647 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 23484
I20260809 04:53:26.424309 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:26.410664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52090 (local address 127.31.93.1:39573)
0809 04:53:26.411199 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:26.411208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:26.411239 (+ 31us) server_negotiation.cc:407] Connection header received
0809 04:53:26.411309 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:26.411318 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:26.411606 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:26.411777 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:26.413787 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:26.414828 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:26.416000 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:26.416542 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:26.416640 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:26.419137 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:26.419149 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:26.419162 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:26.419197 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:26.421712 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:26.422410 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:26.422418 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:26.422420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:26.422583 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:26.422919 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:26.422922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:26.422924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:26.423290 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:26.423528 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:26.423758 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 04:53:26.423956 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":224,"threads_started":1}
I20260809 04:53:26.425534 23726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:26.428297 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:26.427452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44725 (local address 127.31.93.3:36773)
0809 04:53:26.427975 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:26.428099 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":235,"threads_started":1}
I20260809 04:53:26.512354 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 23484
I20260809 04:53:26.536612 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 23738
I20260809 04:53:26.549320 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:26.537347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.31.93.2:46529)
0809 04:53:26.537823 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:26.537827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:26.537844 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:53:26.537900 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:26.537903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:26.537965 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:26.538063 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:26.539543 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:26.540340 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:26.541505 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:26.541902 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:26.541999 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:26.544628 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:26.544633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:26.544637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:26.544669 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:26.547127 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:26.547714 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:26.547717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:26.547719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:26.547847 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:26.548129 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:26.548132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:26.548134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:26.548370 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:26.548576 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:26.548782 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 04:53:26.548955 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":256,"threads_started":1}
I20260809 04:53:26.549993 23915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:26.629832 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 23738
I20260809 04:53:26.652923 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 23931
I20260809 04:53:26.665692 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:26.653810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60246 (local address 127.31.93.3:34783)
0809 04:53:26.654359 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:26.654363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:26.654381 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:53:26.654434 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:26.654437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:26.654495 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:26.654605 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:26.655959 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:26.656731 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:26.657944 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:26.658422 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:26.658538 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:26.660960 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:26.660965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:26.660969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:26.661001 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:26.663261 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:26.663850 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:26.663856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:26.663859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:26.663972 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:26.664313 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:26.664318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:26.664322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:26.664581 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:26.664822 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:26.665114 (+ 292us) server_negotiation.cc:299] Negotiation successful
0809 04:53:26.665329 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":254,"threads_started":1}
I20260809 04:53:26.666472 24125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:26.746830 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 23931
2026-08-09T04:53:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24190 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:53:26 dist-test-slave-4tdj krb5kdc[24236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:53:26 dist-test-slave-4tdj krb5kdc[24236](info): set up 2 sockets
Aug 09 04:53:26 dist-test-slave-4tdj krb5kdc[24236](info): commencing operation
krb5kdc: starting...
W20260809 04:53:27.970721 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:53:28 dist-test-slave-4tdj krb5kdc[24236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:53:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:53:28.203725 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:36055
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:36055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:53:28.523010 24260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:28.523291 24260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:28.523432 24260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:28.530404 24260 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 04:53:28.530498 24260 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 04:53:28.530572 24260 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 04:53:28.530646 24260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:53:28.530730 24260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:28.530823 24260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:28.530887 24260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:53:28.530959 24260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:53:28.540306 24260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:36055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:36055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:28.541512 24260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:28.542873 24260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:28.554116 24267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:28.554268 24273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:28.554191 24270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:28.554569 24260 server_base.cc:1034] running on GCE node
I20260809 04:53:28.555564 24260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:28.556808 24260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:28.558019 24260 hybrid_clock.cc:648] HybridClock initialized: now 1786251208557943 us; error 42 us; skew 500 ppm
I20260809 04:53:28.558389 24260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:28 dist-test-slave-4tdj krb5kdc[24236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251208, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:28.561352 24260 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:53:28.562728 24260 webserver.cc:460] Webserver started at http://127.31.93.62:44341/ using document root <none> and password file <none>
I20260809 04:53:28.563392 24260 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:28.563491 24260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:28.563794 24260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:53:28.566725 24260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "e7b58870621f46a0885f998b7aff4dd9"
format_stamp: "Formatted at 2026-08-09 04:53:28 on dist-test-slave-4tdj"
server_key: "63e80624708e11b799457a5086b5ca1e"
server_key_iv: "f3d7c69049df0e8a91763a3a44d0ed81"
server_key_version: "encryptionkey@0"
I20260809 04:53:28.567468 24260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "e7b58870621f46a0885f998b7aff4dd9"
format_stamp: "Formatted at 2026-08-09 04:53:28 on dist-test-slave-4tdj"
server_key: "63e80624708e11b799457a5086b5ca1e"
server_key_iv: "f3d7c69049df0e8a91763a3a44d0ed81"
server_key_version: "encryptionkey@0"
I20260809 04:53:28.572269 24260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260809 04:53:28.575580 24282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:28.576864 24260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:53:28.577028 24260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "e7b58870621f46a0885f998b7aff4dd9"
format_stamp: "Formatted at 2026-08-09 04:53:28 on dist-test-slave-4tdj"
server_key: "63e80624708e11b799457a5086b5ca1e"
server_key_iv: "f3d7c69049df0e8a91763a3a44d0ed81"
server_key_version: "encryptionkey@0"
I20260809 04:53:28.577217 24260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:28.610829 24260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:28.614871 24260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:28.615129 24260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:28.632149 24260 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:36055
I20260809 04:53:28.632191 24364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:36055 every 8 connection(s)
I20260809 04:53:28.633430 24260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:53:28.635493 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 24260
I20260809 04:53:28.635697 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:53:28.636333 32116 external_mini_cluster.cc:1428] Setting key 49c22c0e5aa43b9db36f507aac9fe034
I20260809 04:53:28.637074 24368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 04:53:28 dist-test-slave-4tdj krb5kdc[24236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:53:28.650800 24368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Bootstrap starting.
I20260809 04:53:28.654642 24368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Neither blocks nor log segments found. Creating new log.
I20260809 04:53:28.655879 24368 log.cc:824] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:28.658556 24368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: No bootstrap required, opened a new log
I20260809 04:53:28.659389 24371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:28.638180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.31.93.62:36055)
0809 04:53:28.639000 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:28.639020 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:28.639069 (+ 49us) server_negotiation.cc:407] Connection header received
0809 04:53:28.640686 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:28.640713 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:28.641207 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:28.641627 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:28.643011 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:28.644990 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:28.646404 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:28.647386 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:28.647580 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:28.651617 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:28.651627 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:28.651644 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:28.651687 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:28.655421 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:28.656128 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:28.656135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:28.656141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:28.656312 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:28.656711 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:28.656714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:28.656716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:28.657248 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:28.657537 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:28.658003 (+ 466us) server_negotiation.cc:299] Negotiation successful
0809 04:53:28.658412 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":343,"threads_started":1}
I20260809 04:53:28.666383 24368 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } }
I20260809 04:53:28.666720 24368 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:53:28.666808 24368 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7b58870621f46a0885f998b7aff4dd9, State: Initialized, Role: FOLLOWER
I20260809 04:53:28.667299 24368 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } }
I20260809 04:53:28.667532 24368 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:53:28.667658 24368 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:53:28.667801 24368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:53:28.669795 24368 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } }
I20260809 04:53:28.670238 24368 leader_election.cc:302] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7b58870621f46a0885f998b7aff4dd9; no voters:
I20260809 04:53:28.670882 24368 leader_election.cc:288] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:53:28.670984 24374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:53:28.671818 24374 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 1 LEADER]: Becoming Leader. State: Replica: e7b58870621f46a0885f998b7aff4dd9, State: Running, Role: LEADER
I20260809 04:53:28.672312 24374 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } }
I20260809 04:53:28.672832 24368 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:53:28.675325 24377 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7b58870621f46a0885f998b7aff4dd9. Latest consensus state: current_term: 1 leader_uuid: "e7b58870621f46a0885f998b7aff4dd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } } }
I20260809 04:53:28.675359 24375 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7b58870621f46a0885f998b7aff4dd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } } }
I20260809 04:53:28.675750 24377 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [sys.catalog]: This master's current role is: LEADER
I20260809 04:53:28.675758 24375 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [sys.catalog]: This master's current role is: LEADER
I20260809 04:53:28.677567 24384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:53:28.682742 24384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:53:28.690907 24384 catalog_manager.cc:1347] Generated new cluster ID: e5f2b615df844a02b429cf8508c083a3
I20260809 04:53:28.691008 24384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:53:28.704097 24384 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:53:28.705276 24384 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:53:28.719513 24384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Generated new TSK 0
I20260809 04:53:28.720269 24384 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:53:28.784114 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:36055
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:53:29.108917 24406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:29.109198 24406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:29.109335 24406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:29.116430 24406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:29.116545 24406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:29.116699 24406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:53:29.126329 24406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:29.127544 24406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:29.128909 24406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:29.141521 24416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:29.141659 24418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:29.141558 24421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:29.142552 24406 server_base.cc:1034] running on GCE node
I20260809 04:53:29.143118 24406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:29.143965 24406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:29.145277 24406 hybrid_clock.cc:648] HybridClock initialized: now 1786251209145219 us; error 43 us; skew 500 ppm
I20260809 04:53:29.145596 24406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:29 dist-test-slave-4tdj krb5kdc[24236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251209, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:29.148639 24406 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:53:29.149969 24406 webserver.cc:460] Webserver started at http://127.31.93.1:36553/ using document root <none> and password file <none>
I20260809 04:53:29.150655 24406 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:29.150774 24406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:29.151053 24406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:53:29.153822 24406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "303a5bf262f24e038319178ec5f34cdc"
format_stamp: "Formatted at 2026-08-09 04:53:29 on dist-test-slave-4tdj"
server_key: "0e2a13d23a5657daf27e0be8801bea39"
server_key_iv: "6a18c2101238e69dab3a832de873f9a9"
server_key_version: "encryptionkey@0"
I20260809 04:53:29.154582 24406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "303a5bf262f24e038319178ec5f34cdc"
format_stamp: "Formatted at 2026-08-09 04:53:29 on dist-test-slave-4tdj"
server_key: "0e2a13d23a5657daf27e0be8801bea39"
server_key_iv: "6a18c2101238e69dab3a832de873f9a9"
server_key_version: "encryptionkey@0"
I20260809 04:53:29.159353 24406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:53:29.162721 24431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:29.163977 24406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 04:53:29.164114 24406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "303a5bf262f24e038319178ec5f34cdc"
format_stamp: "Formatted at 2026-08-09 04:53:29 on dist-test-slave-4tdj"
server_key: "0e2a13d23a5657daf27e0be8801bea39"
server_key_iv: "6a18c2101238e69dab3a832de873f9a9"
server_key_version: "encryptionkey@0"
I20260809 04:53:29.164278 24406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:29.179353 24406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:29.183452 24406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:29.183744 24406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:29.184660 24406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:53:29.185874 24406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:53:29.185946 24406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:29.186040 24406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:53:29.186084 24406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:29.215245 24406 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:38963
I20260809 04:53:29.215291 24606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:38963 every 8 connection(s)
I20260809 04:53:29.216451 24406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:53:29.218686 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 24406
I20260809 04:53:29.218906 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:53:29.219746 32116 external_mini_cluster.cc:1428] Setting key 240039f8107c7df0d85421c2aa31c013
Aug 09 04:53:29 dist-test-slave-4tdj krb5kdc[24236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251209, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:53:29.236613 24609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:29.218937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:33291 (local address 127.31.93.62:36055)
0809 04:53:29.219546 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:29.219552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:29.220536 (+ 984us) server_negotiation.cc:407] Connection header received
0809 04:53:29.222705 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:29.222710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:29.222786 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:29.222886 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:29.224316 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:29.225265 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:29.226821 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:29.227341 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:29.227457 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:29.230792 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:29.230797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:29.230802 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:29.230838 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:29.233500 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:29.234346 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:29.234351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:29.234354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:29.234501 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:29.234903 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:29.234907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:29.234910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:29.235252 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:29.235388 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:29.236048 (+ 660us) server_negotiation.cc:299] Negotiation successful
0809 04:53:29.236265 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":343,"threads_started":1}
I20260809 04:53:29.237267 24611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:29.219512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:33291)
0809 04:53:29.220356 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:29.220407 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:29.222451 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:29.223088 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:29.223096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:29.223581 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:53:29.224145 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:29.224163 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:29.225386 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:29.225389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:29.226684 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:29.226697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:29.227517 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:29.227522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:29.227707 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:29.228346 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:53:29.228373 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:53:29.230583 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:53:29.233648 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:29.233656 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:29.233670 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:29.234197 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:29.234610 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:29.234613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:29.234615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:29.234797 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:29.235360 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:53:29.235365 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:53:29.235661 (+ 296us) client_negotiation.cc:769] Sending connection context
0809 04:53:29.235864 (+ 203us) client_negotiation.cc:240] Negotiation successful
0809 04:53:29.236213 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":344,"threads_started":1}
I20260809 04:53:29.238772 24607 heartbeater.cc:347] Connected to a master server at 127.31.93.62:36055
I20260809 04:53:29.239080 24607 heartbeater.cc:464] Registering TS with master...
I20260809 04:53:29.239959 24607 heartbeater.cc:511] Master 127.31.93.62:36055 requested a full tablet report, sending...
I20260809 04:53:29.242233 24312 ts_manager.cc:194] Registered new tserver with Master: 303a5bf262f24e038319178ec5f34cdc (127.31.93.1:38963)
I20260809 04:53:29.244185 24312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:33291
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:53:29.277973 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:36055
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:53:29.592844 24615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:29.593082 24615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:29.593178 24615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:29.600033 24615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:29.600124 24615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:29.600236 24615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:53:29.609196 24615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:29.610246 24615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:29.611474 24615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:29.623580 24622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:29.623586 24623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:29.623615 24626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:29.624364 24615 server_base.cc:1034] running on GCE node
I20260809 04:53:29.624933 24615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:29.625763 24615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:29.626951 24615 hybrid_clock.cc:648] HybridClock initialized: now 1786251209626905 us; error 36 us; skew 500 ppm
I20260809 04:53:29.627264 24615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:29 dist-test-slave-4tdj krb5kdc[24236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251209, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:29.633128 24615 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:53:29.634682 24615 webserver.cc:460] Webserver started at http://127.31.93.2:41231/ using document root <none> and password file <none>
I20260809 04:53:29.635465 24615 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:29.635593 24615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:29.635882 24615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:53:29.639464 24615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "372ec9e2a94043b6afa7c582c714a8aa"
format_stamp: "Formatted at 2026-08-09 04:53:29 on dist-test-slave-4tdj"
server_key: "428837f2e2c4b90fd770544fdb7a42e8"
server_key_iv: "e10973d94c39ee77840e850896749500"
server_key_version: "encryptionkey@0"
I20260809 04:53:29.640331 24615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "372ec9e2a94043b6afa7c582c714a8aa"
format_stamp: "Formatted at 2026-08-09 04:53:29 on dist-test-slave-4tdj"
server_key: "428837f2e2c4b90fd770544fdb7a42e8"
server_key_iv: "e10973d94c39ee77840e850896749500"
server_key_version: "encryptionkey@0"
I20260809 04:53:29.646320 24615 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 04:53:29.650266 24638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:29.651676 24615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:53:29.651818 24615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "372ec9e2a94043b6afa7c582c714a8aa"
format_stamp: "Formatted at 2026-08-09 04:53:29 on dist-test-slave-4tdj"
server_key: "428837f2e2c4b90fd770544fdb7a42e8"
server_key_iv: "e10973d94c39ee77840e850896749500"
server_key_version: "encryptionkey@0"
I20260809 04:53:29.651979 24615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:29.672364 24615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:29.675936 24615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:29.676172 24615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:29.677034 24615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:53:29.678226 24615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:53:29.678325 24615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:29.678427 24615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:53:29.678471 24615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:29.707638 24615 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:43375
I20260809 04:53:29.707700 24816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:43375 every 8 connection(s)
I20260809 04:53:29.708782 24615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:53:29.709774 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 24615
I20260809 04:53:29.709961 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:53:29.710532 32116 external_mini_cluster.cc:1428] Setting key 68a21dd8c8ee9325fd5a7e65f15068c2
Aug 09 04:53:29 dist-test-slave-4tdj krb5kdc[24236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251209, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:53:29.727929 24609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:29.711392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:34899 (local address 127.31.93.62:36055)
0809 04:53:29.711572 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:29.711576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:29.712857 (+ 1281us) server_negotiation.cc:407] Connection header received
0809 04:53:29.714662 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:29.714668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:29.714747 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:29.714828 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:29.716213 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:29.716913 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:29.718453 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:29.718875 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:29.718983 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:29.722095 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:29.722100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:29.722104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:29.722135 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:29.724723 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:29.725517 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:29.725521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:29.725530 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:29.725679 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:29.726113 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:29.726116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:29.726118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:29.726412 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:29.726538 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:29.727330 (+ 792us) server_negotiation.cc:299] Negotiation successful
0809 04:53:29.727583 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:53:29.728972 24819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:29.711896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:34899)
0809 04:53:29.712708 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:29.712758 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:29.714425 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:29.715060 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:29.715067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:29.715539 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:53:29.716056 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:29.716072 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:29.717054 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:29.717058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:29.718331 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:29.718343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:29.719086 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:29.719093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:29.719312 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:29.719878 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:53:29.719908 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:53:29.721889 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:53:29.724848 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:29.724857 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:29.724872 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:29.725393 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:29.725783 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:29.725788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:29.725792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:29.726017 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:29.726528 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:53:29.726535 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:53:29.726941 (+ 406us) client_negotiation.cc:769] Sending connection context
0809 04:53:29.727266 (+ 325us) client_negotiation.cc:240] Negotiation successful
0809 04:53:29.727726 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":310,"threads_started":1}
I20260809 04:53:29.730351 24817 heartbeater.cc:347] Connected to a master server at 127.31.93.62:36055
I20260809 04:53:29.730659 24817 heartbeater.cc:464] Registering TS with master...
I20260809 04:53:29.731478 24817 heartbeater.cc:511] Master 127.31.93.62:36055 requested a full tablet report, sending...
I20260809 04:53:29.732981 24312 ts_manager.cc:194] Registered new tserver with Master: 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375)
I20260809 04:53:29.733773 24312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:34899
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:53:29.769608 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:36055
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:53:30.085139 24823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:30.085412 24823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:30.085527 24823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:30.092566 24823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:30.092664 24823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:30.092819 24823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:53:30.101987 24823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:30.103153 24823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:30.104367 24823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:30.116655 24831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:30.116583 24833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:30.116629 24835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:30.117389 24823 server_base.cc:1034] running on GCE node
I20260809 04:53:30.117949 24823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:30.118837 24823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:30.120038 24823 hybrid_clock.cc:648] HybridClock initialized: now 1786251210119999 us; error 27 us; skew 500 ppm
I20260809 04:53:30.120333 24823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:30 dist-test-slave-4tdj krb5kdc[24236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251210, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:30.123252 24823 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:53:30.124595 24823 webserver.cc:460] Webserver started at http://127.31.93.3:45215/ using document root <none> and password file <none>
I20260809 04:53:30.125236 24823 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:30.125352 24823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:30.125624 24823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:53:30.128429 24823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "34ba07b555464b4eb9783460041cd7db"
format_stamp: "Formatted at 2026-08-09 04:53:30 on dist-test-slave-4tdj"
server_key: "4619ee9cf0546af9dd304c76aec32437"
server_key_iv: "2ca18714654e6376cd4d0be6f9ffaba1"
server_key_version: "encryptionkey@0"
I20260809 04:53:30.129142 24823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "34ba07b555464b4eb9783460041cd7db"
format_stamp: "Formatted at 2026-08-09 04:53:30 on dist-test-slave-4tdj"
server_key: "4619ee9cf0546af9dd304c76aec32437"
server_key_iv: "2ca18714654e6376cd4d0be6f9ffaba1"
server_key_version: "encryptionkey@0"
I20260809 04:53:30.133937 24823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260809 04:53:30.137099 24848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:30.138401 24823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:53:30.138545 24823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "34ba07b555464b4eb9783460041cd7db"
format_stamp: "Formatted at 2026-08-09 04:53:30 on dist-test-slave-4tdj"
server_key: "4619ee9cf0546af9dd304c76aec32437"
server_key_iv: "2ca18714654e6376cd4d0be6f9ffaba1"
server_key_version: "encryptionkey@0"
I20260809 04:53:30.138707 24823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:30.165127 24823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:30.168772 24823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:30.169015 24823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:30.169884 24823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:53:30.171190 24823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:53:30.171268 24823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:30.171382 24823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:53:30.171437 24823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:30.201674 24823 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:34407
I20260809 04:53:30.201717 25020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:34407 every 8 connection(s)
I20260809 04:53:30.202859 24823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:53:30.211217 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 24823
I20260809 04:53:30.211392 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:53:30.211833 32116 external_mini_cluster.cc:1428] Setting key 6c33c4b6da7e40d3f71a665c84e90e1d
Aug 09 04:53:30 dist-test-slave-4tdj krb5kdc[24236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251210, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:53:30.222028 24609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.205169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:50299 (local address 127.31.93.62:36055)
0809 04:53:30.205347 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:30.205351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:30.206464 (+ 1113us) server_negotiation.cc:407] Connection header received
0809 04:53:30.208194 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:30.208198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:30.208281 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:30.208366 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:30.209724 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.210492 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.212022 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.212559 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.212663 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.216164 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:30.216171 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:30.216175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:30.216214 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:30.219382 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:30.220017 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:30.220023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:30.220027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:30.220186 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:30.220680 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:30.220683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:30.220685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:30.220940 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:30.221049 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:30.221645 (+ 596us) server_negotiation.cc:299] Negotiation successful
0809 04:53:30.221793 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 04:53:30.222965 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.205586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:50299)
0809 04:53:30.206277 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:30.206343 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:30.207960 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:30.208562 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:30.208570 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:30.209050 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:53:30.209595 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:30.209612 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.210613 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.210617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:30.211871 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:30.211884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.212779 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.212786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:30.213008 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.213671 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:53:30.213699 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:53:30.215963 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:53:30.219503 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:30.219510 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:30.219519 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:30.219901 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:30.220328 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:30.220332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:30.220335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:30.220563 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:30.221038 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:53:30.221043 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:53:30.221341 (+ 298us) client_negotiation.cc:769] Sending connection context
0809 04:53:30.221588 (+ 247us) client_negotiation.cc:240] Negotiation successful
0809 04:53:30.221932 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":264,"threads_started":1}
I20260809 04:53:30.224236 25021 heartbeater.cc:347] Connected to a master server at 127.31.93.62:36055
I20260809 04:53:30.224524 25021 heartbeater.cc:464] Registering TS with master...
I20260809 04:53:30.225214 25021 heartbeater.cc:511] Master 127.31.93.62:36055 requested a full tablet report, sending...
I20260809 04:53:30.226763 24312 ts_manager.cc:194] Registered new tserver with Master: 34ba07b555464b4eb9783460041cd7db (127.31.93.3:34407)
I20260809 04:53:30.227537 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:53:30.227643 24312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:50299
I20260809 04:53:30.244665 24609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.232866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.31.93.62:36055)
0809 04:53:30.233007 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:30.233011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:30.233482 (+ 471us) server_negotiation.cc:407] Connection header received
0809 04:53:30.233619 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:30.233622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:30.233671 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:30.233741 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:30.235264 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.235898 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.237296 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.237672 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.237754 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.240143 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:30.240147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:30.240151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:30.240178 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:30.242434 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:30.243161 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:30.243165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:30.243167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:30.243284 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:30.243605 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:30.243607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:30.243609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:30.243889 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:30.244064 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:30.244314 (+ 250us) server_negotiation.cc:299] Negotiation successful
0809 04:53:30.244454 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 04:53:30.247622 24607 heartbeater.cc:503] Master 127.31.93.62:36055 was elected leader, sending a full tablet report...
I20260809 04:53:30.248409 24312 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:53:30.255095 24312 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:53:30.275808 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.266383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.31.93.2:43375)
0809 04:53:30.267022 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:30.267031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:30.267312 (+ 281us) server_negotiation.cc:407] Connection header received
0809 04:53:30.267410 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:30.267416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:30.267644 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:30.267827 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:30.268662 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.270260 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.273189 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.274676 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.274795 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.274982 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:30.275342 (+ 360us) server_negotiation.cc:299] Negotiation successful
0809 04:53:30.275494 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":740,"threads_started":1}
I20260809 04:53:30.275889 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.266069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:43375 (local address 127.0.0.1:58302)
0809 04:53:30.266997 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:30.267034 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:30.267199 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:30.268142 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:30.268150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:30.268191 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:30.268519 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:30.268531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.270438 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.270443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:30.273025 (+ 2582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:30.273046 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.274881 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.274889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:30.275099 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.275156 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 04:53:30.275273 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 04:53:30.275458 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":28,"thread_start_us":542,"threads_started":1}
I20260809 04:53:30.276167 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.267366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:38963 (local address 127.0.0.1:46062)
0809 04:53:30.268004 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:30.268021 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:30.268123 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:30.269301 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:30.269306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:30.269327 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:30.269641 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:30.269652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.271334 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.271339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:30.273372 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:30.273383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.275152 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.275157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:30.275309 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.275341 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 04:53:30.275426 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 04:53:30.275549 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":21,"thread_start_us":377,"threads_started":1}
I20260809 04:53:30.277623 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.268058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:34407 (local address 127.0.0.1:37308)
0809 04:53:30.268988 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:30.269006 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:30.269120 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:30.270966 (+ 1846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:30.270971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:30.270996 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:30.271483 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:30.271504 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.272869 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.272874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:30.275036 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:30.275051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.276839 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.276845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:30.277004 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.277028 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:53:30.277108 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 04:53:30.277223 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":3,"thread_start_us":834,"threads_started":1}
I20260809 04:53:30.277632 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.268619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.31.93.3:34407)
0809 04:53:30.270474 (+ 1855us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:30.270482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:30.270508 (+ 26us) server_negotiation.cc:407] Connection header received
0809 04:53:30.270599 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:30.270605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:30.270796 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:30.270979 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:30.271661 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.272745 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.275216 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.276662 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.276784 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.276983 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:30.277184 (+ 201us) server_negotiation.cc:299] Negotiation successful
0809 04:53:30.277330 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1703,"thread_start_us":1000,"threads_started":1}
I20260809 04:53:30.279450 24720 tablet_service.cc:1467] Processing CreateTablet for tablet ffe5e7e3d9744978ba820f822638d30d (DEFAULT_TABLE table=test-table [id=c8f47013f8cf414ba03741ce0a83f2b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:53:30.280649 24720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe5e7e3d9744978ba820f822638d30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:53:30.280669 24920 tablet_service.cc:1467] Processing CreateTablet for tablet ffe5e7e3d9744978ba820f822638d30d (DEFAULT_TABLE table=test-table [id=c8f47013f8cf414ba03741ce0a83f2b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:53:30.281790 24920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe5e7e3d9744978ba820f822638d30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:53:30.282840 25042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.267667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.31.93.1:38963)
0809 04:53:30.268683 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:30.268692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:30.268719 (+ 27us) server_negotiation.cc:407] Connection header received
0809 04:53:30.268807 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:30.268815 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:30.269050 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:30.269292 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:30.269800 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.271212 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.273554 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.274814 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.274931 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.275154 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:30.276564 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0809 04:53:30.276718 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":697,"threads_started":1}
I20260809 04:53:30.286056 24510 tablet_service.cc:1467] Processing CreateTablet for tablet ffe5e7e3d9744978ba820f822638d30d (DEFAULT_TABLE table=test-table [id=c8f47013f8cf414ba03741ce0a83f2b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:53:30.287554 24510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe5e7e3d9744978ba820f822638d30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:53:30.292047 25048 tablet_bootstrap.cc:492] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa: Bootstrap starting.
I20260809 04:53:30.295073 25048 tablet_bootstrap.cc:654] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa: Neither blocks nor log segments found. Creating new log.
I20260809 04:53:30.295540 25049 tablet_bootstrap.cc:492] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db: Bootstrap starting.
I20260809 04:53:30.296898 25048 log.cc:824] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:30.298791 25049 tablet_bootstrap.cc:654] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db: Neither blocks nor log segments found. Creating new log.
I20260809 04:53:30.300001 25049 log.cc:824] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:30.300061 25048 tablet_bootstrap.cc:492] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa: No bootstrap required, opened a new log
I20260809 04:53:30.300348 25048 ts_tablet_manager.cc:1397] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260809 04:53:30.302945 25052 tablet_bootstrap.cc:492] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc: Bootstrap starting.
I20260809 04:53:30.305541 25049 tablet_bootstrap.cc:492] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db: No bootstrap required, opened a new log
I20260809 04:53:30.305799 25049 ts_tablet_manager.cc:1397] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260809 04:53:30.306735 25052 tablet_bootstrap.cc:654] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc: Neither blocks nor log segments found. Creating new log.
I20260809 04:53:30.307621 25052 log.cc:824] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:30.309845 25052 tablet_bootstrap.cc:492] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc: No bootstrap required, opened a new log
I20260809 04:53:30.310076 25052 ts_tablet_manager.cc:1397] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260809 04:53:30.310206 25048 raft_consensus.cc:348] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } }
I20260809 04:53:30.310649 25048 raft_consensus.cc:374] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:53:30.310767 25048 raft_consensus.cc:729] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372ec9e2a94043b6afa7c582c714a8aa, State: Initialized, Role: FOLLOWER
I20260809 04:53:30.311468 25048 consensus_queue.cc:260] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } }
I20260809 04:53:30.312546 24817 heartbeater.cc:503] Master 127.31.93.62:36055 was elected leader, sending a full tablet report...
I20260809 04:53:30.313150 25048 ts_tablet_manager.cc:1428] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260809 04:53:30.315572 25049 raft_consensus.cc:348] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } }
I20260809 04:53:30.315982 25049 raft_consensus.cc:374] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:53:30.316097 25049 raft_consensus.cc:729] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ba07b555464b4eb9783460041cd7db, State: Initialized, Role: FOLLOWER
I20260809 04:53:30.316733 25049 consensus_queue.cc:260] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } }
I20260809 04:53:30.317703 25021 heartbeater.cc:503] Master 127.31.93.62:36055 was elected leader, sending a full tablet report...
I20260809 04:53:30.318336 25049 ts_tablet_manager.cc:1428] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260809 04:53:30.319077 25052 raft_consensus.cc:348] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } }
I20260809 04:53:30.319479 25052 raft_consensus.cc:374] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:53:30.319586 25052 raft_consensus.cc:729] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303a5bf262f24e038319178ec5f34cdc, State: Initialized, Role: FOLLOWER
I20260809 04:53:30.320166 25052 consensus_queue.cc:260] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } }
I20260809 04:53:30.321765 25052 ts_tablet_manager.cc:1428] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260809 04:53:30.360087 25061 raft_consensus.cc:482] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:53:30.360369 25061 raft_consensus.cc:504] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } }
I20260809 04:53:30.362006 25061 leader_election.cc:288] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375), 303a5bf262f24e038319178ec5f34cdc (127.31.93.1:38963)
I20260809 04:53:30.367468 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.362384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:43375 (local address 127.31.93.3:36229)
0809 04:53:30.362591 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:30.362612 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:30.362739 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:30.363080 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:30.363085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:30.363113 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:30.363310 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:30.363322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.364318 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.364322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:30.365645 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:30.365657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.366783 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.366788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:30.366953 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.366978 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 04:53:30.367063 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 04:53:30.367190 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":1}
I20260809 04:53:30.367481 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.362388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:36229 (local address 127.31.93.2:43375)
0809 04:53:30.362757 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:30.362763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:30.362783 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:53:30.362854 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:30.362860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:30.362932 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:30.363026 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:30.363444 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.364172 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.365798 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.366610 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.366729 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.366818 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:30.367107 (+ 289us) server_negotiation.cc:299] Negotiation successful
0809 04:53:30.367224 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260809 04:53:30.368359 24752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe5e7e3d9744978ba820f822638d30d" candidate_uuid: "34ba07b555464b4eb9783460041cd7db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372ec9e2a94043b6afa7c582c714a8aa" is_pre_election: true
I20260809 04:53:30.368836 24752 raft_consensus.cc:2454] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34ba07b555464b4eb9783460041cd7db in term 0.
I20260809 04:53:30.368927 25042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.363171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:58181 (local address 127.31.93.1:38963)
0809 04:53:30.363355 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:30.363360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:30.364001 (+ 641us) server_negotiation.cc:407] Connection header received
0809 04:53:30.364144 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:30.364148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:30.364204 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:30.364275 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:30.364830 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.365514 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.367135 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.368092 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.368225 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.368316 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:30.368575 (+ 259us) server_negotiation.cc:299] Negotiation successful
0809 04:53:30.368701 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 04:53:30.368995 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.362790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:38963 (local address 127.31.93.3:58181)
0809 04:53:30.363902 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:30.363915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:30.364025 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:30.364303 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:30.364308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:30.364331 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:30.364684 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:30.364692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.365669 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.365672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:30.366971 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:30.366983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.368284 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.368290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:30.368438 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.368461 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:53:30.368534 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 04:53:30.368653 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"thread_start_us":822,"threads_started":1}
I20260809 04:53:30.369445 24855 leader_election.cc:302] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34ba07b555464b4eb9783460041cd7db, 372ec9e2a94043b6afa7c582c714a8aa; no voters:
I20260809 04:53:30.369823 25061 raft_consensus.cc:2781] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:53:30.369973 25061 raft_consensus.cc:482] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:53:30.370088 25061 raft_consensus.cc:3037] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:53:30.369971 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe5e7e3d9744978ba820f822638d30d" candidate_uuid: "34ba07b555464b4eb9783460041cd7db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303a5bf262f24e038319178ec5f34cdc" is_pre_election: true
I20260809 04:53:30.370488 24540 raft_consensus.cc:2454] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34ba07b555464b4eb9783460041cd7db in term 0.
I20260809 04:53:30.372575 25061 raft_consensus.cc:504] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } }
I20260809 04:53:30.373148 25061 leader_election.cc:288] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [CANDIDATE]: Term 1 election: Requested vote from peers 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375), 303a5bf262f24e038319178ec5f34cdc (127.31.93.1:38963)
I20260809 04:53:30.373610 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe5e7e3d9744978ba820f822638d30d" candidate_uuid: "34ba07b555464b4eb9783460041cd7db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303a5bf262f24e038319178ec5f34cdc"
I20260809 04:53:30.373611 24752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe5e7e3d9744978ba820f822638d30d" candidate_uuid: "34ba07b555464b4eb9783460041cd7db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372ec9e2a94043b6afa7c582c714a8aa"
I20260809 04:53:30.373768 24540 raft_consensus.cc:3037] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:53:30.373780 24752 raft_consensus.cc:3037] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:53:30.375872 24540 raft_consensus.cc:2454] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34ba07b555464b4eb9783460041cd7db in term 1.
I20260809 04:53:30.375891 24752 raft_consensus.cc:2454] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34ba07b555464b4eb9783460041cd7db in term 1.
I20260809 04:53:30.376220 24855 leader_election.cc:302] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34ba07b555464b4eb9783460041cd7db, 372ec9e2a94043b6afa7c582c714a8aa; no voters:
I20260809 04:53:30.376518 25061 raft_consensus.cc:2781] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:53:30.376946 25061 raft_consensus.cc:686] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [term 1 LEADER]: Becoming Leader. State: Replica: 34ba07b555464b4eb9783460041cd7db, State: Running, Role: LEADER
I20260809 04:53:30.377386 25061 consensus_queue.cc:237] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } }
I20260809 04:53:30.381918 24312 catalog_manager.cc:5495] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db reported cstate change: term changed from 0 to 1, leader changed from <none> to 34ba07b555464b4eb9783460041cd7db (127.31.93.3). New cstate: current_term: 1 leader_uuid: "34ba07b555464b4eb9783460041cd7db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } health_report { overall_health: HEALTHY } } }
I20260809 04:53:30.398139 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.393222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.31.93.3:34407)
0809 04:53:30.393392 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:30.393396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:30.393447 (+ 51us) server_negotiation.cc:407] Connection header received
0809 04:53:30.393656 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:30.393660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:30.393748 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:30.393875 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:53:30.394174 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.394912 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.396299 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.396696 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.396790 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.397246 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:53:30.397740 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:53:30.397853 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:30.397891 (+ 38us) server_negotiation.cc:299] Negotiation successful
0809 04:53:30.397953 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 04:53:30.405665 24752 raft_consensus.cc:1261] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa [term 1 FOLLOWER]: Refusing update from remote peer 34ba07b555464b4eb9783460041cd7db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:53:30.405668 24540 raft_consensus.cc:1261] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Refusing update from remote peer 34ba07b555464b4eb9783460041cd7db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:53:30.406512 25064 consensus_queue.cc:1035] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [LEADER]: Connected to new peer: Peer: permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:53:30.406816 25061 consensus_queue.cc:1035] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [LEADER]: Connected to new peer: Peer: permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:53:30.417348 25069 mvcc.cc:205] Tried to move back new op lower bound from 7316484957812953088 to 7316484957709484032. Current Snapshot: MvccSnapshot[applied={T|T < 7316484957812953088}]
I20260809 04:53:30.428941 24312 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:53:30.429152 24312 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:53:30.432530 24312 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Sending DeleteTablet for 3 replicas of tablet ffe5e7e3d9744978ba820f822638d30d
I20260809 04:53:30.433525 24720 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe5e7e3d9744978ba820f822638d30d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:53:30 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:58302
I20260809 04:53:30.433770 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 24406
I20260809 04:53:30.433717 24920 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe5e7e3d9744978ba820f822638d30d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:53:30 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:37308
I20260809 04:53:30.433946 24510 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe5e7e3d9744978ba820f822638d30d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:53:30 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:46062
I20260809 04:53:30.434710 25082 tablet_replica.cc:330] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa: stopping tablet replica
I20260809 04:53:30.435111 25082 raft_consensus.cc:2229] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:53:30.435580 25082 raft_consensus.cc:2258] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:53:30.437753 25082 ts_tablet_manager.cc:1905] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:53:30.444800 25082 ts_tablet_manager.cc:1918] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:53:30.444962 25082 log.cc:1196] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/ffe5e7e3d9744978ba820f822638d30d
I20260809 04:53:30.445504 25082 ts_tablet_manager.cc:1939] T ffe5e7e3d9744978ba820f822638d30d P 372ec9e2a94043b6afa7c582c714a8aa: Deleting consensus metadata
I20260809 04:53:30.447237 24290 catalog_manager.cc:4841] TS 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375): tablet ffe5e7e3d9744978ba820f822638d30d (table test-table [id=c8f47013f8cf414ba03741ce0a83f2b8]) successfully deleted
I20260809 04:53:30.449838 25084 tablet_replica.cc:330] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc: stopping tablet replica
I20260809 04:53:30.450266 25084 raft_consensus.cc:2229] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:53:30.450726 25084 raft_consensus.cc:2258] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:53:30.451363 25083 tablet_replica.cc:330] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db: stopping tablet replica
I20260809 04:53:30.451676 25083 raft_consensus.cc:2229] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:53:30.452164 25083 raft_consensus.cc:2258] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:53:30.453719 25083 ts_tablet_manager.cc:1905] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 04:53:30 dist-test-slave-4tdj krb5kdc[24236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:53:30.455793 25084 ts_tablet_manager.cc:1905] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:53:30.461335 25083 ts_tablet_manager.cc:1918] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:53:30.461508 25083 log.cc:1196] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/ffe5e7e3d9744978ba820f822638d30d
I20260809 04:53:30.462057 25083 ts_tablet_manager.cc:1939] T ffe5e7e3d9744978ba820f822638d30d P 34ba07b555464b4eb9783460041cd7db: Deleting consensus metadata
I20260809 04:53:30.463147 25084 ts_tablet_manager.cc:1918] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:53:30.463312 25084 log.cc:1196] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/ffe5e7e3d9744978ba820f822638d30d
I20260809 04:53:30.463769 24290 catalog_manager.cc:4841] TS 34ba07b555464b4eb9783460041cd7db (127.31.93.3:34407): tablet ffe5e7e3d9744978ba820f822638d30d (table test-table [id=c8f47013f8cf414ba03741ce0a83f2b8]) successfully deleted
I20260809 04:53:30.463861 25042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.435044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46066 (local address 127.31.93.1:38963)
0809 04:53:30.435214 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:30.435220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:30.444109 (+ 8889us) server_negotiation.cc:407] Connection header received
0809 04:53:30.444523 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:30.444529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:30.444654 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:30.444740 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:30.446988 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.447944 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.449409 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.450074 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.450191 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.455903 (+ 5712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:30.455917 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:30.455933 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:30.455971 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:30.459813 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:30.460779 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:30.460792 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:30.460796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:30.460951 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:30.461615 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:30.461621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:30.461624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:30.462165 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:30.462507 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:30.463147 (+ 640us) server_negotiation.cc:299] Negotiation successful
0809 04:53:30.463478 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 04:53:30.463889 25084 ts_tablet_manager.cc:1939] T ffe5e7e3d9744978ba820f822638d30d P 303a5bf262f24e038319178ec5f34cdc: Deleting consensus metadata
I20260809 04:53:30.465190 24599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:30.465935 24290 catalog_manager.cc:4841] TS 303a5bf262f24e038319178ec5f34cdc (127.31.93.1:38963): tablet ffe5e7e3d9744978ba820f822638d30d (table test-table [id=c8f47013f8cf414ba03741ce0a83f2b8]) successfully deleted
I20260809 04:53:30.554577 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 24406
I20260809 04:53:30.577114 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 24615
Aug 09 04:53:30 dist-test-slave-4tdj krb5kdc[24236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:53:30.592352 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.577969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58306 (local address 127.31.93.2:43375)
0809 04:53:30.578143 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:30.578149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:30.578217 (+ 68us) server_negotiation.cc:407] Connection header received
0809 04:53:30.578425 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:30.578432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:30.578507 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:30.578602 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:30.579895 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.580733 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.582205 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.582661 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.582772 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.587097 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:30.587111 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:30.587126 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:30.587165 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:30.589757 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:30.590381 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:30.590389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:30.590391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:30.590514 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:30.590886 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:30.590889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:30.590891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:30.591251 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:30.591535 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:30.591846 (+ 311us) server_negotiation.cc:299] Negotiation successful
0809 04:53:30.592048 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 04:53:30.592978 24811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:30.693240 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 24615
I20260809 04:53:30.715186 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 24823
Aug 09 04:53:30 dist-test-slave-4tdj krb5kdc[24236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:53:30.732731 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:30.715931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.31.93.3:34407)
0809 04:53:30.716124 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:30.716130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:30.716151 (+ 21us) server_negotiation.cc:407] Connection header received
0809 04:53:30.716267 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:30.716272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:30.716362 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:30.716475 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:30.717700 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.718924 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.720206 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:30.720886 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:30.721020 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:30.725514 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:30.725530 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:30.725551 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:30.725599 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:30.729393 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:30.730215 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:30.730225 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:30.730228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:30.730663 (+ 435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:30.731063 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:30.731068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:30.731071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:30.731547 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:30.731861 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:30.732077 (+ 216us) server_negotiation.cc:299] Negotiation successful
0809 04:53:30.732298 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 04:53:30.733645 25013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:30.852283 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 24823
I20260809 04:53:30.874720 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 24260
I20260809 04:53:30.875770 24357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:30.983062 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 24260
I20260809 04:53:51.008220 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:36055
--webserver_interface=127.31.93.62
--webserver_port=44341
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:36055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:53:51.341028 25092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:51.341300 25092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:51.341427 25092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:51.348349 25092 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 04:53:51.348434 25092 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 04:53:51.348497 25092 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 04:53:51.348563 25092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:53:51.348644 25092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:51.348732 25092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:51.348793 25092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:53:51.348866 25092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:53:51.358227 25092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:36055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:36055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=44341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:51.359390 25092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:51.360693 25092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:51.383551 25102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:51.383633 25100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:51.383663 25099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:51.384233 25092 server_base.cc:1034] running on GCE node
I20260809 04:53:51.385070 25092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:51.386533 25092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:51.387750 25092 hybrid_clock.cc:648] HybridClock initialized: now 1786251231387687 us; error 31 us; skew 500 ppm
I20260809 04:53:51.388084 25092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:51 dist-test-slave-4tdj krb5kdc[24236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251231, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:51.391484 25092 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:53:51.392942 25092 webserver.cc:460] Webserver started at http://127.31.93.62:44341/ using document root <none> and password file <none>
I20260809 04:53:51.393648 25092 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:51.393750 25092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:51.399341 25092 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260809 04:53:51.402109 25113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:51.403429 25092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:53:51.403590 25092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "e7b58870621f46a0885f998b7aff4dd9"
format_stamp: "Formatted at 2026-08-09 04:53:28 on dist-test-slave-4tdj"
server_key: "63e80624708e11b799457a5086b5ca1e"
server_key_iv: "f3d7c69049df0e8a91763a3a44d0ed81"
server_key_version: "encryptionkey@0"
I20260809 04:53:51.404556 25092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:51.432469 25092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:51.436342 25092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:51.436601 25092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:51.455917 25092 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:36055
I20260809 04:53:51.455950 25189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:36055 every 8 connection(s)
I20260809 04:53:51.457381 25092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:53:51.461283 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 25092
I20260809 04:53:51.462067 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:38963
--local_ip_for_outbound_sockets=127.31.93.1
--tserver_master_addrs=127.31.93.62:36055
--webserver_port=36553
--webserver_interface=127.31.93.1
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:53:51.464401 25191 sys_catalog.cc:263] Verifying existing consensus state
I20260809 04:53:51.467694 25191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Bootstrap starting.
I20260809 04:53:51.486970 25191 log.cc:824] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:51.498986 25191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:53:51.499456 25191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Bootstrap complete.
I20260809 04:53:51.508354 25191 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } }
I20260809 04:53:51.509238 25191 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7b58870621f46a0885f998b7aff4dd9, State: Initialized, Role: FOLLOWER
I20260809 04:53:51.509789 25191 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } }
I20260809 04:53:51.510020 25191 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:53:51.510139 25191 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:53:51.510324 25191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 04:53:51.513909 25191 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } }
I20260809 04:53:51.514431 25191 leader_election.cc:302] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7b58870621f46a0885f998b7aff4dd9; no voters:
I20260809 04:53:51.515031 25191 leader_election.cc:288] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 04:53:51.515201 25195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 04:53:51.516319 25195 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 2 LEADER]: Becoming Leader. State: Replica: e7b58870621f46a0885f998b7aff4dd9, State: Running, Role: LEADER
I20260809 04:53:51.516916 25195 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } }
I20260809 04:53:51.517206 25191 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:53:51.519385 25197 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7b58870621f46a0885f998b7aff4dd9. Latest consensus state: current_term: 2 leader_uuid: "e7b58870621f46a0885f998b7aff4dd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } } }
I20260809 04:53:51.519784 25197 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [sys.catalog]: This master's current role is: LEADER
I20260809 04:53:51.520319 25196 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e7b58870621f46a0885f998b7aff4dd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } } }
I20260809 04:53:51.520617 25196 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [sys.catalog]: This master's current role is: LEADER
I20260809 04:53:51.525290 25202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:53:51.533705 25202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:53:51.535231 25202 catalog_manager.cc:1259] Loaded cluster ID: e5f2b615df844a02b429cf8508c083a3
I20260809 04:53:51.535734 25202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:53:51.539899 25202 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:53:51.541544 25202 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Loaded TSK: 0
I20260809 04:53:51.550096 25218 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 04:53:51.566521 25202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Generated new TSK 1
I20260809 04:53:51.567006 25202 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 04:53:51.819945 25193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:51.820192 25193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:51.820312 25193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:51.827306 25193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:51.827402 25193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:51.827534 25193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:53:51.837119 25193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:38963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=36553
--tserver_master_addrs=127.31.93.62:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:51.838353 25193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:51.839658 25193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:51.853793 25228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:51.853819 25229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:51.854815 25231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:51.855312 25193 server_base.cc:1034] running on GCE node
I20260809 04:53:51.855859 25193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:51.856766 25193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:51.857967 25193 hybrid_clock.cc:648] HybridClock initialized: now 1786251231857911 us; error 49 us; skew 500 ppm
I20260809 04:53:51.858259 25193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:51 dist-test-slave-4tdj krb5kdc[24236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251231, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:51.861299 25193 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:53:51.862728 25193 webserver.cc:460] Webserver started at http://127.31.93.1:36553/ using document root <none> and password file <none>
I20260809 04:53:51.863368 25193 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:51.863495 25193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:51.868877 25193 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:53:51.871614 25241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:51.872944 25193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 04:53:51.873106 25193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "303a5bf262f24e038319178ec5f34cdc"
format_stamp: "Formatted at 2026-08-09 04:53:29 on dist-test-slave-4tdj"
server_key: "0e2a13d23a5657daf27e0be8801bea39"
server_key_iv: "6a18c2101238e69dab3a832de873f9a9"
server_key_version: "encryptionkey@0"
I20260809 04:53:51.874011 25193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:51.894838 25193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:51.898699 25193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:51.898939 25193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:51.899881 25193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:53:51.901139 25193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:53:51.901247 25193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:51.901352 25193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:53:51.901438 25193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:51.929962 25193 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:38963
I20260809 04:53:51.930032 25412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:38963 every 8 connection(s)
I20260809 04:53:51.931416 25193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:53:51.938069 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 25193
I20260809 04:53:51.938874 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:43375
--local_ip_for_outbound_sockets=127.31.93.2
--tserver_master_addrs=127.31.93.62:36055
--webserver_port=41231
--webserver_interface=127.31.93.2
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:53:51 dist-test-slave-4tdj krb5kdc[24236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251231, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:53:51.954845 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:51.934166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:52491 (local address 127.31.93.62:36055)
0809 04:53:51.934711 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:51.934723 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:51.935537 (+ 814us) server_negotiation.cc:407] Connection header received
0809 04:53:51.937727 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:51.937746 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:51.938211 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:51.938620 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:51.940128 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:51.941825 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:51.943353 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:51.944293 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:51.944467 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:51.947642 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:51.947652 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:51.947669 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:51.947712 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:51.950631 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:51.951321 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:51.951328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:51.951334 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:51.951493 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:51.951901 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:51.951904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:51.951906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:51.952339 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:51.952523 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:51.953273 (+ 750us) server_negotiation.cc:299] Negotiation successful
0809 04:53:51.953649 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":244,"threads_started":1}
I20260809 04:53:51.954841 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:51.934464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:52491)
0809 04:53:51.935330 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:51.935388 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:51.937152 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:51.938725 (+ 1573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:51.938732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:51.939327 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:53:51.939910 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:51.939934 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:51.941981 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:51.941985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:51.943216 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:51.943229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:51.944483 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:51.944487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:51.944674 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:51.945292 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:53:51.945320 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:53:51.947377 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:53:51.950761 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:51.950769 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:51.950783 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:51.951184 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:51.951603 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:51.951608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:51.951612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:51.951787 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:51.952476 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:53:51.952483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:53:51.952796 (+ 313us) client_negotiation.cc:769] Sending connection context
0809 04:53:51.953025 (+ 229us) client_negotiation.cc:240] Negotiation successful
0809 04:53:51.953544 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":327,"threads_started":1}
I20260809 04:53:51.957618 25413 heartbeater.cc:347] Connected to a master server at 127.31.93.62:36055
I20260809 04:53:51.957888 25413 heartbeater.cc:464] Registering TS with master...
I20260809 04:53:51.958631 25413 heartbeater.cc:511] Master 127.31.93.62:36055 requested a full tablet report, sending...
I20260809 04:53:51.960703 25134 ts_manager.cc:194] Registered new tserver with Master: 303a5bf262f24e038319178ec5f34cdc (127.31.93.1:38963)
I20260809 04:53:51.962172 25134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:52491
W20260809 04:53:52.258015 25419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:52.258265 25419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:52.258419 25419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:52.265206 25419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:52.265328 25419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:52.265472 25419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:53:52.274638 25419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:43375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=41231
--tserver_master_addrs=127.31.93.62:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:52.275774 25419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:52.276960 25419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:52.289083 25432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:52.289166 25427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:52.289135 25429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:52.289837 25419 server_base.cc:1034] running on GCE node
I20260809 04:53:52.290474 25419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:52.291320 25419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:52.292532 25419 hybrid_clock.cc:648] HybridClock initialized: now 1786251232292482 us; error 37 us; skew 500 ppm
I20260809 04:53:52.292811 25419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:52 dist-test-slave-4tdj krb5kdc[24236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251232, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:52.295702 25419 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:53:52.296959 25419 webserver.cc:460] Webserver started at http://127.31.93.2:41231/ using document root <none> and password file <none>
I20260809 04:53:52.297576 25419 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:52.297704 25419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:52.302681 25419 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:53:52.305159 25445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:52.306388 25419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:53:52.306555 25419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "372ec9e2a94043b6afa7c582c714a8aa"
format_stamp: "Formatted at 2026-08-09 04:53:29 on dist-test-slave-4tdj"
server_key: "428837f2e2c4b90fd770544fdb7a42e8"
server_key_iv: "e10973d94c39ee77840e850896749500"
server_key_version: "encryptionkey@0"
I20260809 04:53:52.307489 25419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:52.332832 25419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:52.336656 25419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:52.336890 25419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:52.337762 25419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:53:52.339017 25419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:53:52.339123 25419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:52.339226 25419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:53:52.339293 25419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:52.368171 25419 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:43375
I20260809 04:53:52.368258 25614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:43375 every 8 connection(s)
I20260809 04:53:52.369508 25419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:53:52.374541 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 25419
I20260809 04:53:52.375200 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:34407
--local_ip_for_outbound_sockets=127.31.93.3
--tserver_master_addrs=127.31.93.62:36055
--webserver_port=45215
--webserver_interface=127.31.93.3
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 04:53:52 dist-test-slave-4tdj krb5kdc[24236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251232, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:53:52.388873 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:52.371965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:52023 (local address 127.31.93.62:36055)
0809 04:53:52.372147 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:52.372151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:52.373241 (+ 1090us) server_negotiation.cc:407] Connection header received
0809 04:53:52.375060 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:52.375067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:52.375154 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:52.375279 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:52.376699 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.377629 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.379283 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.379836 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.379956 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:52.383169 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:52.383175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:52.383179 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:52.383219 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:52.386022 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:52.386749 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:52.386755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:52.386758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:52.386877 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:52.387270 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:52.387276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:52.387279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:52.387536 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:52.387674 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:52.388301 (+ 627us) server_negotiation.cc:299] Negotiation successful
0809 04:53:52.388524 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 04:53:52.389696 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:52.372402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:52023)
0809 04:53:52.373081 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:52.373121 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:52.374803 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:52.375498 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:52.375509 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:52.375985 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:53:52.376540 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:52.376556 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.377773 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.377778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:52.379127 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:52.379143 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.380016 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.380020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:52.380213 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:52.380799 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:53:52.380820 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:53:52.382942 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:53:52.386147 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:52.386155 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:52.386169 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:52.386628 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:52.386974 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:52.386979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:52.386983 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:52.387157 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:52.387662 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:53:52.387670 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:53:52.387950 (+ 280us) client_negotiation.cc:769] Sending connection context
0809 04:53:52.388175 (+ 225us) client_negotiation.cc:240] Negotiation successful
0809 04:53:52.388627 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":260,"threads_started":1}
I20260809 04:53:52.391058 25615 heartbeater.cc:347] Connected to a master server at 127.31.93.62:36055
I20260809 04:53:52.391368 25615 heartbeater.cc:464] Registering TS with master...
I20260809 04:53:52.392141 25615 heartbeater.cc:511] Master 127.31.93.62:36055 requested a full tablet report, sending...
I20260809 04:53:52.393703 25134 ts_manager.cc:194] Registered new tserver with Master: 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375)
I20260809 04:53:52.394799 25134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:52023
W20260809 04:53:52.700728 25619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:53:52.700973 25619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:53:52.701076 25619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:53:52.707965 25619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:53:52.708072 25619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:53:52.708210 25619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:53:52.717674 25619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:34407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=45215
--tserver_master_addrs=127.31.93.62:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:53:52.718972 25619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:53:52.720260 25619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:53:52.732494 25624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:52.732542 25627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:53:52.732597 25625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:53:52.733496 25619 server_base.cc:1034] running on GCE node
I20260809 04:53:52.734017 25619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:53:52.734954 25619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:53:52.736200 25619 hybrid_clock.cc:648] HybridClock initialized: now 1786251232736148 us; error 58 us; skew 500 ppm
I20260809 04:53:52.736584 25619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:53:52 dist-test-slave-4tdj krb5kdc[24236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251232, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:53:52.739637 25619 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:53:52.741072 25619 webserver.cc:460] Webserver started at http://127.31.93.3:45215/ using document root <none> and password file <none>
I20260809 04:53:52.741698 25619 fs_manager.cc:359] Metadata directory not provided
I20260809 04:53:52.741829 25619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:53:52.747047 25619 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 04:53:52.749686 25641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:52.751031 25619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:53:52.751194 25619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "34ba07b555464b4eb9783460041cd7db"
format_stamp: "Formatted at 2026-08-09 04:53:30 on dist-test-slave-4tdj"
server_key: "4619ee9cf0546af9dd304c76aec32437"
server_key_iv: "2ca18714654e6376cd4d0be6f9ffaba1"
server_key_version: "encryptionkey@0"
I20260809 04:53:52.752190 25619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:53:52.777915 25619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:53:52.782101 25619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:53:52.782416 25619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:53:52.783434 25619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:53:52.784785 25619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:53:52.784900 25619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:52.785019 25619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:53:52.785097 25619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:53:52.816061 25619 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:34407
I20260809 04:53:52.816128 25814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:34407 every 8 connection(s)
I20260809 04:53:52.817425 25619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:53:52.823442 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 25619
Aug 09 04:53:52 dist-test-slave-4tdj krb5kdc[24236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251232, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:53:52.836614 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:52.819768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:49065 (local address 127.31.93.62:36055)
0809 04:53:52.819930 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:52.819934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:52.821037 (+ 1103us) server_negotiation.cc:407] Connection header received
0809 04:53:52.822826 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:52.822832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:52.822909 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:52.822993 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:52.824537 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.825474 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.827001 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.827460 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.827565 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:52.830790 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:52.830795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:52.830799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:52.830828 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:52.833540 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:52.834394 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:52.834398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:52.834400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:52.834523 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:52.834971 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:52.834974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:52.834976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:52.835248 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:52.835401 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:52.836019 (+ 618us) server_negotiation.cc:299] Negotiation successful
0809 04:53:52.836239 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":52}
I20260809 04:53:52.837515 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:52.820211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:49065)
0809 04:53:52.820891 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:52.820927 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:52.822572 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:52.823247 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:52.823255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:52.823749 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:53:52.824335 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:52.824360 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.825643 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.825646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:52.826855 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:52.826867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.827665 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.827673 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:52.827902 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:52.828463 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:53:52.828484 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:53:52.830609 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:53:52.833669 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:52.833677 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:52.833691 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:52.834251 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:52.834626 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:53:52.834631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:53:52.834635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:53:52.834858 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:53:52.835382 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:53:52.835389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:53:52.835667 (+ 278us) client_negotiation.cc:769] Sending connection context
0809 04:53:52.835914 (+ 247us) client_negotiation.cc:240] Negotiation successful
0809 04:53:52.836447 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":259,"threads_started":1}
I20260809 04:53:52.837715 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:52.824354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.31.93.62:36055)
0809 04:53:52.825098 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:52.825104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:52.825120 (+ 16us) server_negotiation.cc:407] Connection header received
0809 04:53:52.825167 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:52.825170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:52.825231 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:52.825317 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:52.826841 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.827646 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.828938 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.829335 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.829437 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:52.832164 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:52.832168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:52.832171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:52.832210 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:52.835129 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:52.835812 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:52.835818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:52.835822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:52.835930 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:52.836300 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:52.836306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:52.836309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:52.836613 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:52.836847 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:52.837141 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 04:53:52.837339 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":623,"thread_start_us":502,"threads_started":1}
I20260809 04:53:52.838865 25817 heartbeater.cc:347] Connected to a master server at 127.31.93.62:36055
I20260809 04:53:52.839143 25817 heartbeater.cc:464] Registering TS with master...
I20260809 04:53:52.839936 25817 heartbeater.cc:511] Master 127.31.93.62:36055 requested a full tablet report, sending...
I20260809 04:53:52.841198 25133 ts_manager.cc:194] Registered new tserver with Master: 34ba07b555464b4eb9783460041cd7db (127.31.93.3:34407)
I20260809 04:53:52.841966 25133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:49065
I20260809 04:53:52.852007 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:53:52.857995 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:52.852798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.31.93.62:36055)
0809 04:53:52.852939 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:52.852944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:52.853358 (+ 414us) server_negotiation.cc:407] Connection header received
0809 04:53:52.853537 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:52.853541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:52.853649 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:52.853766 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:53:52.854185 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.854861 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.856191 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.856608 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.856710 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:52.857183 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:53:52.857420 (+ 237us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0809 04:53:52.857735 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52652: token expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260809 04:53:52.858098 25825 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52652: token expired
W20260809 04:53:52.858609 25026 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.93.62:36055)
I20260809 04:53:52.870064 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:52.859006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.31.93.62:36055)
0809 04:53:52.859143 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:52.859148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:52.859188 (+ 40us) server_negotiation.cc:407] Connection header received
0809 04:53:52.859329 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:52.859334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:52.859382 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:52.859454 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:52.860696 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.861336 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.862678 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.863053 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.863165 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:52.865475 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:52.865480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:52.865483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:52.865532 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:52.867623 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:52.868243 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:52.868247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:52.868249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:52.868367 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:52.868715 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:52.868720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:52.868723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:52.869058 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:52.869278 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:52.869584 (+ 306us) server_negotiation.cc:299] Negotiation successful
0809 04:53:52.869777 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 04:53:52.873013 25133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:53:52.880105 25133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:53:52.902735 25833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:52.890876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.31.93.2:43375)
0809 04:53:52.891545 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:52.891551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:52.895004 (+ 3453us) server_negotiation.cc:407] Connection header received
0809 04:53:52.895324 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:52.895339 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:52.895596 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:52.896508 (+ 912us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:52.896719 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.898181 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.900310 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.901521 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.901651 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:52.901844 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:52.902253 (+ 409us) server_negotiation.cc:299] Negotiation successful
0809 04:53:52.902437 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":451,"threads_started":1}
I20260809 04:53:52.902799 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:52.890912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:43375 (local address 127.0.0.1:56390)
0809 04:53:52.894835 (+ 3923us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:52.894884 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:52.895058 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:52.895833 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:52.895840 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:52.895871 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:52.896283 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:52.896298 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.898471 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.898477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:52.900166 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:52.900177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.901765 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.901773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:52.901997 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:52.902051 (+ 54us) client_negotiation.cc:769] Sending connection context
0809 04:53:52.902181 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 04:53:52.902376 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3690,"mutex_wait_us":40,"thread_start_us":3520,"threads_started":1}
I20260809 04:53:52.905663 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:52.896116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46328 (local address 127.31.93.1:38963)
0809 04:53:52.896661 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:52.896669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:52.897296 (+ 627us) server_negotiation.cc:407] Connection header received
0809 04:53:52.897571 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:52.897580 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:52.897803 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:52.897984 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:52.899095 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.900606 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.903427 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.904645 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.904773 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:52.904941 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:52.905204 (+ 263us) server_negotiation.cc:299] Negotiation successful
0809 04:53:52.905381 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":244,"threads_started":1}
I20260809 04:53:52.905683 25840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:52.896657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:38963 (local address 127.0.0.1:46328)
0809 04:53:52.897191 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:52.897208 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:52.897393 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:52.898227 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:52.898231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:52.898254 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:52.898902 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:52.898917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.900783 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.900791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:52.903227 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:52.903240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.904840 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.904846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:52.905016 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:52.905047 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 04:53:52.905138 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 04:53:52.905267 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":171,"thread_start_us":275,"threads_started":1}
I20260809 04:53:52.906217 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:52.895271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:34407 (local address 127.0.0.1:42464)
0809 04:53:52.895806 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:52.895825 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:52.895941 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:52.898096 (+ 2155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:52.898101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:52.898126 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:52.898667 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:52.898679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.900739 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.900744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:52.903681 (+ 2937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:52.903695 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.905354 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.905360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:52.905535 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:52.905580 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 04:53:52.905658 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 04:53:52.905893 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":82,"thread_start_us":293,"threads_started":1}
I20260809 04:53:52.906598 25524 tablet_service.cc:1467] Processing CreateTablet for tablet 5e58313bc4894e4d96c124ff753aefa4 (DEFAULT_TABLE table=test-table [id=f12922634eac423cb9dfeb573bc2fcac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:53:52.907194 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:52.895430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42464 (local address 127.31.93.3:34407)
0809 04:53:52.897516 (+ 2086us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:52.897523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:52.897604 (+ 81us) server_negotiation.cc:407] Connection header received
0809 04:53:52.897700 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:52.897707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:52.897904 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:52.898818 (+ 914us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:52.899048 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.900616 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.903832 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:52.905111 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:52.906531 (+ 1420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:52.906692 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:52.906741 (+ 49us) server_negotiation.cc:299] Negotiation successful
0809 04:53:52.906870 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1926,"thread_start_us":1774,"threads_started":1}
I20260809 04:53:52.907961 25524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e58313bc4894e4d96c124ff753aefa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:53:52.909258 25309 tablet_service.cc:1467] Processing CreateTablet for tablet 5e58313bc4894e4d96c124ff753aefa4 (DEFAULT_TABLE table=test-table [id=f12922634eac423cb9dfeb573bc2fcac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:53:52.910365 25309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e58313bc4894e4d96c124ff753aefa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:53:52.910717 25711 tablet_service.cc:1467] Processing CreateTablet for tablet 5e58313bc4894e4d96c124ff753aefa4 (DEFAULT_TABLE table=test-table [id=f12922634eac423cb9dfeb573bc2fcac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:53:52.912029 25711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e58313bc4894e4d96c124ff753aefa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:53:52.921630 25841 tablet_bootstrap.cc:492] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc: Bootstrap starting.
I20260809 04:53:52.923497 25842 tablet_bootstrap.cc:492] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa: Bootstrap starting.
I20260809 04:53:52.925800 25841 tablet_bootstrap.cc:654] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc: Neither blocks nor log segments found. Creating new log.
I20260809 04:53:52.926589 25843 tablet_bootstrap.cc:492] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db: Bootstrap starting.
I20260809 04:53:52.927083 25842 tablet_bootstrap.cc:654] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa: Neither blocks nor log segments found. Creating new log.
I20260809 04:53:52.928242 25842 log.cc:824] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:52.928242 25841 log.cc:824] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:52.929750 25843 tablet_bootstrap.cc:654] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db: Neither blocks nor log segments found. Creating new log.
I20260809 04:53:52.930788 25843 log.cc:824] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db: Log is configured to *not* fsync() on all Append() calls
I20260809 04:53:52.936966 25842 tablet_bootstrap.cc:492] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa: No bootstrap required, opened a new log
I20260809 04:53:52.937269 25842 ts_tablet_manager.cc:1397] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260809 04:53:52.939659 25843 tablet_bootstrap.cc:492] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db: No bootstrap required, opened a new log
I20260809 04:53:52.939908 25843 ts_tablet_manager.cc:1397] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260809 04:53:52.944501 25841 tablet_bootstrap.cc:492] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc: No bootstrap required, opened a new log
I20260809 04:53:52.944794 25841 ts_tablet_manager.cc:1397] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.009s
I20260809 04:53:52.946125 25842 raft_consensus.cc:348] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } }
I20260809 04:53:52.946509 25842 raft_consensus.cc:374] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:53:52.946617 25842 raft_consensus.cc:729] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372ec9e2a94043b6afa7c582c714a8aa, State: Initialized, Role: FOLLOWER
I20260809 04:53:52.947176 25842 consensus_queue.cc:260] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } }
I20260809 04:53:52.948187 25615 heartbeater.cc:503] Master 127.31.93.62:36055 was elected leader, sending a full tablet report...
I20260809 04:53:52.948819 25842 ts_tablet_manager.cc:1428] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260809 04:53:52.949713 25843 raft_consensus.cc:348] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } }
I20260809 04:53:52.950130 25843 raft_consensus.cc:374] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:53:52.950224 25843 raft_consensus.cc:729] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ba07b555464b4eb9783460041cd7db, State: Initialized, Role: FOLLOWER
I20260809 04:53:52.950814 25843 consensus_queue.cc:260] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } }
I20260809 04:53:52.951908 25817 heartbeater.cc:503] Master 127.31.93.62:36055 was elected leader, sending a full tablet report...
I20260809 04:53:52.952487 25843 ts_tablet_manager.cc:1428] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db: Time spent starting tablet: real 0.012s user 0.010s sys 0.003s
I20260809 04:53:52.954566 25841 raft_consensus.cc:348] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } }
I20260809 04:53:52.954977 25841 raft_consensus.cc:374] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:53:52.955120 25841 raft_consensus.cc:729] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303a5bf262f24e038319178ec5f34cdc, State: Initialized, Role: FOLLOWER
I20260809 04:53:52.955796 25841 consensus_queue.cc:260] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } }
I20260809 04:53:52.956853 25413 heartbeater.cc:503] Master 127.31.93.62:36055 was elected leader, sending a full tablet report...
I20260809 04:53:52.957439 25841 ts_tablet_manager.cc:1428] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260809 04:53:53.067140 25854 raft_consensus.cc:482] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:53:53.067425 25854 raft_consensus.cc:504] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } }
W20260809 04:53:53.069033 25820 tablet.cc:2369] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:53:53.069186 25854 leader_election.cc:288] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375), 34ba07b555464b4eb9783460041cd7db (127.31.93.3:34407)
I20260809 04:53:53.076206 25833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:53.069603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:41077 (local address 127.31.93.2:43375)
0809 04:53:53.069830 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:53.069836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:53.070603 (+ 767us) server_negotiation.cc:407] Connection header received
0809 04:53:53.070912 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:53.070917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:53.070983 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:53.071031 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:53.071837 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:53.072630 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:53.074453 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:53.075225 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:53.075330 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:53.075407 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:53.075816 (+ 409us) server_negotiation.cc:299] Negotiation successful
0809 04:53:53.075941 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:53:53.076208 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:53.070686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:59867 (local address 127.31.93.3:34407)
0809 04:53:53.070878 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:53.070883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:53.071212 (+ 329us) server_negotiation.cc:407] Connection header received
0809 04:53:53.071347 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:53.071352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:53.071426 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:53.071511 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:53:53.072101 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:53.072809 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:53.074513 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:53.075324 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:53.075454 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:53.075551 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:53.075862 (+ 311us) server_negotiation.cc:299] Negotiation successful
0809 04:53:53.075977 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 04:53:53.076327 25857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:53.070552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:34407 (local address 127.31.93.1:59867)
0809 04:53:53.071115 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:53.071129 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:53.071233 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:53.071545 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:53.071548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:53.071579 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:53.071953 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:53.071964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:53.072950 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:53.072953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:53.074363 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:53.074376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:53.075541 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:53.075545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:53.075709 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:53.075733 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:53:53.075818 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 04:53:53.075941 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":17,"thread_start_us":276,"threads_started":1}
I20260809 04:53:53.076735 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:53.069860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:43375 (local address 127.31.93.1:41077)
0809 04:53:53.070482 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:53.070501 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:53:53.070732 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:53:53.071199 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:53:53.071203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:53:53.071298 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:53:53.071680 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:53.071691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:53.072764 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:53.072770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:53.074275 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:53:53.074314 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:53:53.075446 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:53:53.075453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:53:53.075650 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:53.075693 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 04:53:53.075763 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 04:53:53.075885 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":5,"spinlock_wait_cycles":24448,"thread_start_us":312,"threads_started":1}
I20260809 04:53:53.077301 25744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e58313bc4894e4d96c124ff753aefa4" candidate_uuid: "303a5bf262f24e038319178ec5f34cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34ba07b555464b4eb9783460041cd7db" is_pre_election: true
I20260809 04:53:53.077654 25553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e58313bc4894e4d96c124ff753aefa4" candidate_uuid: "303a5bf262f24e038319178ec5f34cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372ec9e2a94043b6afa7c582c714a8aa" is_pre_election: true
I20260809 04:53:53.077874 25744 raft_consensus.cc:2454] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 303a5bf262f24e038319178ec5f34cdc in term 0.
I20260809 04:53:53.078183 25553 raft_consensus.cc:2454] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 303a5bf262f24e038319178ec5f34cdc in term 0.
I20260809 04:53:53.078622 25247 leader_election.cc:302] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 303a5bf262f24e038319178ec5f34cdc, 34ba07b555464b4eb9783460041cd7db; no voters:
I20260809 04:53:53.078962 25854 raft_consensus.cc:2781] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:53:53.079106 25854 raft_consensus.cc:482] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:53:53.079202 25854 raft_consensus.cc:3037] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:53:53.081580 25854 raft_consensus.cc:504] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } }
I20260809 04:53:53.082093 25854 leader_election.cc:288] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 1 election: Requested vote from peers 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375), 34ba07b555464b4eb9783460041cd7db (127.31.93.3:34407)
I20260809 04:53:53.082517 25553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e58313bc4894e4d96c124ff753aefa4" candidate_uuid: "303a5bf262f24e038319178ec5f34cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372ec9e2a94043b6afa7c582c714a8aa"
I20260809 04:53:53.082578 25744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e58313bc4894e4d96c124ff753aefa4" candidate_uuid: "303a5bf262f24e038319178ec5f34cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34ba07b555464b4eb9783460041cd7db"
I20260809 04:53:53.082672 25553 raft_consensus.cc:3037] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:53:53.082767 25744 raft_consensus.cc:3037] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:53:53.084777 25553 raft_consensus.cc:2454] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 303a5bf262f24e038319178ec5f34cdc in term 1.
I20260809 04:53:53.084822 25744 raft_consensus.cc:2454] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 303a5bf262f24e038319178ec5f34cdc in term 1.
I20260809 04:53:53.085273 25247 leader_election.cc:302] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 303a5bf262f24e038319178ec5f34cdc, 34ba07b555464b4eb9783460041cd7db; no voters:
I20260809 04:53:53.085525 25854 raft_consensus.cc:2781] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:53:53.085922 25854 raft_consensus.cc:686] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 LEADER]: Becoming Leader. State: Replica: 303a5bf262f24e038319178ec5f34cdc, State: Running, Role: LEADER
I20260809 04:53:53.086387 25854 consensus_queue.cc:237] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } }
I20260809 04:53:53.090539 25134 catalog_manager.cc:5495] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 303a5bf262f24e038319178ec5f34cdc (127.31.93.1). New cstate: current_term: 1 leader_uuid: "303a5bf262f24e038319178ec5f34cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } health_report { overall_health: HEALTHY } } }
W20260809 04:53:53.121383 25616 tablet.cc:2369] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:53:53.137907 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:53.133173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46342 (local address 127.31.93.1:38963)
0809 04:53:53.133334 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:53.133339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:53.133356 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:53:53.133528 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:53.133532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:53.133629 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:53.133760 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:53:53.134026 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:53.134767 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:53.136111 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:53.136507 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:53.136600 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:53.137009 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:53:53.137470 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:53:53.137598 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:53.137633 (+ 35us) server_negotiation.cc:299] Negotiation successful
0809 04:53:53.137695 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 04:53:53.145630 25744 raft_consensus.cc:1261] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db [term 1 FOLLOWER]: Refusing update from remote peer 303a5bf262f24e038319178ec5f34cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:53:53.145663 25553 raft_consensus.cc:1261] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa [term 1 FOLLOWER]: Refusing update from remote peer 303a5bf262f24e038319178ec5f34cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:53:53.146528 25858 consensus_queue.cc:1035] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:53:53.146852 25854 consensus_queue.cc:1035] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:53:53.169412 25134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:53:53.169706 25134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:53:53.172748 25134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Sending DeleteTablet for 3 replicas of tablet 5e58313bc4894e4d96c124ff753aefa4
I20260809 04:53:53.173755 25524 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e58313bc4894e4d96c124ff753aefa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:53:53 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:56390
I20260809 04:53:53.173904 25711 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e58313bc4894e4d96c124ff753aefa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:53:53 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:42464
I20260809 04:53:53.174182 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 25193
I20260809 04:53:53.174788 25309 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e58313bc4894e4d96c124ff753aefa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:53:53 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:46328
I20260809 04:53:53.183712 25880 tablet_replica.cc:330] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db: stopping tablet replica
I20260809 04:53:53.184114 25880 raft_consensus.cc:2229] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:53:53.184610 25880 raft_consensus.cc:2258] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:53:53.188578 25880 ts_tablet_manager.cc:1905] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:53:53.190918 25881 tablet_replica.cc:330] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc: stopping tablet replica
I20260809 04:53:53.191306 25881 raft_consensus.cc:2229] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:53:53.192003 25881 raft_consensus.cc:2258] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:53:53.194000 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:53.175954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.31.93.1:38963)
0809 04:53:53.176113 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:53.176118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:53.176135 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:53:53.176183 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:53.176187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:53.176267 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:53.176366 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:53.178395 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:53.179688 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:53.181526 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:53.182189 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:53.182335 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:53.186111 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:53.186124 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:53.186140 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:53.186177 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:53.190348 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:53.191296 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:53.191307 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:53.191311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:53.191561 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:53.192063 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:53.192072 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:53.192075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:53.192587 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:53.193083 (+ 496us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:53.193247 (+ 164us) server_negotiation.cc:299] Negotiation successful
0809 04:53:53.193464 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 04:53:53.194685 25407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:53.195500 25879 tablet_replica.cc:330] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa: stopping tablet replica
I20260809 04:53:53.195914 25879 raft_consensus.cc:2229] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:53:53.196424 25879 raft_consensus.cc:2258] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:53:53.196516 25880 ts_tablet_manager.cc:1918] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:53:53.196660 25880 log.cc:1196] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/5e58313bc4894e4d96c124ff753aefa4
I20260809 04:53:53.197247 25880 ts_tablet_manager.cc:1939] T 5e58313bc4894e4d96c124ff753aefa4 P 34ba07b555464b4eb9783460041cd7db: Deleting consensus metadata
I20260809 04:53:53.198379 25879 ts_tablet_manager.cc:1905] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:53:53.199241 25117 catalog_manager.cc:4841] TS 34ba07b555464b4eb9783460041cd7db (127.31.93.3:34407): tablet 5e58313bc4894e4d96c124ff753aefa4 (table test-table [id=f12922634eac423cb9dfeb573bc2fcac]) successfully deleted
I20260809 04:53:53.203809 25879 ts_tablet_manager.cc:1918] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:53:53.203934 25879 log.cc:1196] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/5e58313bc4894e4d96c124ff753aefa4
I20260809 04:53:53.204329 25879 ts_tablet_manager.cc:1939] T 5e58313bc4894e4d96c124ff753aefa4 P 372ec9e2a94043b6afa7c582c714a8aa: Deleting consensus metadata
I20260809 04:53:53.205670 25117 catalog_manager.cc:4841] TS 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375): tablet 5e58313bc4894e4d96c124ff753aefa4 (table test-table [id=f12922634eac423cb9dfeb573bc2fcac]) successfully deleted
I20260809 04:53:53.195724 25881 ts_tablet_manager.cc:1905] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:53:53.289767 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 25193
W20260809 04:53:53.314157 25117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.93.1:38963 (error 108)
I20260809 04:53:53.314239 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 25419
I20260809 04:53:53.316097 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:53.315627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:38963 (local address 127.0.0.1:46358)
0809 04:53:53.315808 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:53.315920 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:38963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 04:53:53.316337 25117 catalog_manager.cc:4584] TS 303a5bf262f24e038319178ec5f34cdc (127.31.93.1:38963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e58313bc4894e4d96c124ff753aefa4: Network error: Client connection negotiation failed: client connection to 127.31.93.1:38963: connect: Connection refused (error 111)
I20260809 04:53:53.327775 25833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:53.315028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.31.93.2:43375)
0809 04:53:53.315209 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:53.315213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:53.315306 (+ 93us) server_negotiation.cc:407] Connection header received
0809 04:53:53.315532 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:53.315537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:53.315626 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:53.315728 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:53.317274 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:53.318021 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:53.319485 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:53.319887 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:53.319983 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:53.322466 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:53.322480 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:53.322494 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:53.322530 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:53.325145 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:53.325813 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:53.325821 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:53.325823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:53.325942 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:53.326312 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:53.326318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:53.326321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:53.326681 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:53.326973 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:53.327273 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 04:53:53.327472 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:53:53.328429 25608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:53.369782 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:53.369293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:38963 (local address 127.0.0.1:46374)
0809 04:53:53.369485 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:53.369590 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:38963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 04:53:53.370896 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:53.370549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:38963 (local address 127.0.0.1:46390)
0809 04:53:53.370704 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:53.370784 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:38963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 04:53:53.414258 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 25419
I20260809 04:53:53.431239 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:53.430799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:38963 (local address 127.0.0.1:46398)
0809 04:53:53.431003 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:53.431091 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:38963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 04:53:53.432231 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:53.431929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:38963 (local address 127.0.0.1:46410)
0809 04:53:53.432048 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:53.432122 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:38963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 04:53:53.435153 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 25619
I20260809 04:53:53.451992 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:53.439825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.31.93.3:34407)
0809 04:53:53.439985 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 04:53:53.439990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:53:53.440016 (+ 26us) server_negotiation.cc:407] Connection header received
0809 04:53:53.440107 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:53:53.440111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:53:53.440176 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:53:53.440266 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:53:53.441557 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:53.442259 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:53.443535 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:53:53.443925 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:53:53.444025 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:53:53.446435 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:53:53.446453 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:53:53.446475 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:53:53.446515 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:53:53.449345 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:53.449995 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:53.450011 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:53.450013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:53.450148 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:53:53.450501 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:53:53.450506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:53:53.450508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:53:53.450869 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:53:53.451180 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0809 04:53:53.451458 (+ 278us) server_negotiation.cc:299] Negotiation successful
0809 04:53:53.451686 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:53:53.452553 25808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:53.513113 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:53.512627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:38963 (local address 127.0.0.1:46424)
0809 04:53:53.512833 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:53.512923 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:38963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260809 04:53:53.514276 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:53:53.513869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:38963 (local address 127.0.0.1:46426)
0809 04:53:53.514074 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0809 04:53:53.514169 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:38963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260809 04:53:53.542104 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 25619
I20260809 04:53:53.563582 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 25092
I20260809 04:53:53.564707 25183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:53:53.674413 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 25092
I20260809 04:54:13.703258 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:36055
--webserver_interface=127.31.93.62
--webserver_port=44341
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:36055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:14.034236 25893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:14.034520 25893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:14.034634 25893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:14.041574 25893 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 04:54:14.041659 25893 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 04:54:14.041702 25893 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 04:54:14.041771 25893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:54:14.041833 25893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:14.041925 25893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:14.041980 25893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:54:14.042063 25893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:54:14.051640 25893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:36055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:36055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=44341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:14.052867 25893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:14.054145 25893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:14.065284 25908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:14.065346 25903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:14.065527 25901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:14.066228 25893 server_base.cc:1034] running on GCE node
I20260809 04:54:14.066974 25893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:14.068224 25893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:14.069442 25893 hybrid_clock.cc:648] HybridClock initialized: now 1786251254069376 us; error 32 us; skew 500 ppm
I20260809 04:54:14.069770 25893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:14 dist-test-slave-4tdj krb5kdc[24236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251254, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:14.078387 25893 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:54:14.080084 25893 webserver.cc:460] Webserver started at http://127.31.93.62:44341/ using document root <none> and password file <none>
I20260809 04:54:14.080803 25893 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:14.080892 25893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:14.086092 25893 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:54:14.088822 25916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:14.090042 25893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:54:14.090214 25893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "e7b58870621f46a0885f998b7aff4dd9"
format_stamp: "Formatted at 2026-08-09 04:53:28 on dist-test-slave-4tdj"
server_key: "63e80624708e11b799457a5086b5ca1e"
server_key_iv: "f3d7c69049df0e8a91763a3a44d0ed81"
server_key_version: "encryptionkey@0"
I20260809 04:54:14.091172 25893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:14.111380 25893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:14.114897 25893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:14.115154 25893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:14.132182 25893 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:36055
I20260809 04:54:14.132220 25985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:36055 every 8 connection(s)
I20260809 04:54:14.133559 25893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:54:14.140033 25986 sys_catalog.cc:263] Verifying existing consensus state
I20260809 04:54:14.140172 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 25893
I20260809 04:54:14.141077 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:38963
--local_ip_for_outbound_sockets=127.31.93.1
--tserver_master_addrs=127.31.93.62:36055
--webserver_port=36553
--webserver_interface=127.31.93.1
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:54:14.143306 25986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Bootstrap starting.
I20260809 04:54:14.163102 25986 log.cc:824] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:14.179086 25986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:54:14.179551 25986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Bootstrap complete.
I20260809 04:54:14.188246 25986 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } }
I20260809 04:54:14.189100 25986 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7b58870621f46a0885f998b7aff4dd9, State: Initialized, Role: FOLLOWER
I20260809 04:54:14.189662 25986 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } }
I20260809 04:54:14.189877 25986 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:54:14.189982 25986 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:54:14.190148 25986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 04:54:14.193753 25986 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } }
I20260809 04:54:14.194242 25986 leader_election.cc:302] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7b58870621f46a0885f998b7aff4dd9; no voters:
I20260809 04:54:14.194919 25986 leader_election.cc:288] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 04:54:14.195060 25997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 04:54:14.196219 25997 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [term 3 LEADER]: Becoming Leader. State: Replica: e7b58870621f46a0885f998b7aff4dd9, State: Running, Role: LEADER
I20260809 04:54:14.196760 25997 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } }
I20260809 04:54:14.197022 25986 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:54:14.199208 25999 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7b58870621f46a0885f998b7aff4dd9. Latest consensus state: current_term: 3 leader_uuid: "e7b58870621f46a0885f998b7aff4dd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } } }
I20260809 04:54:14.199582 25999 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:14.200021 25998 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e7b58870621f46a0885f998b7aff4dd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b58870621f46a0885f998b7aff4dd9" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 36055 } } }
I20260809 04:54:14.200305 25998 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9 [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:14.203588 26006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:54:14.213147 26006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:54:14.214643 26006 catalog_manager.cc:1259] Loaded cluster ID: e5f2b615df844a02b429cf8508c083a3
I20260809 04:54:14.214748 26006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:54:14.218839 26006 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:54:14.220494 26006 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Loaded TSK: 0
I20260809 04:54:14.220638 26006 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e7b58870621f46a0885f998b7aff4dd9: Loaded TSK: 1
F20260809 04:54:14.221719 26006 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f78bfbb89ad google::LogMessage::Fail() at ??:0
@ 0x7f78bfbbb88a google::LogMessage::SendToLog() at ??:0
@ 0x7f78bfbb84de google::LogMessage::Flush() at ??:0
@ 0x7f78bfbb9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f78c391c7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f78c390b73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f78d55e71c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f78d55e6783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f78d56722a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f78d56720c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f78d56bc5e9 std::function<>::operator()() at ??:0
@ 0x7f78d55eacdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f78d55e8c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f78d5670610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f78d5670439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f78c2804345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f78c280791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f78c2807799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f78c27e2eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f78c5abc6db start_thread at ??:0
@ 0x7f78bea5261f clone at ??:0
W20260809 04:54:14.541261 25991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:14.541546 25991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:14.541649 25991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:14.548655 25991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:14.548751 25991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:14.548866 25991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:54:14.558081 25991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:38963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=36553
--tserver_master_addrs=127.31.93.62:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:14.559269 25991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:14.560499 25991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:14.572566 26056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:14.572599 26057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:14.572630 26059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:14.573560 25991 server_base.cc:1034] running on GCE node
I20260809 04:54:14.574141 25991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:14.575026 25991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:14.576210 25991 hybrid_clock.cc:648] HybridClock initialized: now 1786251254576164 us; error 38 us; skew 500 ppm
I20260809 04:54:14.576507 25991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:14 dist-test-slave-4tdj krb5kdc[24236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251254, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:14.579516 25991 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:54:14.580732 25991 webserver.cc:460] Webserver started at http://127.31.93.1:36553/ using document root <none> and password file <none>
I20260809 04:54:14.581316 25991 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:14.581413 25991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:14.586225 25991 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:54:14.588943 26073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:14.590233 25991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 04:54:14.590430 25991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "303a5bf262f24e038319178ec5f34cdc"
format_stamp: "Formatted at 2026-08-09 04:53:29 on dist-test-slave-4tdj"
server_key: "0e2a13d23a5657daf27e0be8801bea39"
server_key_iv: "6a18c2101238e69dab3a832de873f9a9"
server_key_version: "encryptionkey@0"
I20260809 04:54:14.591419 25991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:14.625880 25991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:14.629753 25991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:14.630019 25991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:14.631069 25991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:14.632881 26082 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260809 04:54:14.638280 25991 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 04:54:14.638430 25991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260809 04:54:14.638554 25991 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 04:54:14.641741 25991 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 04:54:14.641822 25991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260809 04:54:14.642127 26082 tablet_bootstrap.cc:492] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc: Bootstrap starting.
I20260809 04:54:14.649802 26082 log.cc:824] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:14.663239 26082 tablet_bootstrap.cc:492] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 04:54:14.663949 26082 tablet_bootstrap.cc:492] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc: Bootstrap complete.
I20260809 04:54:14.664458 26082 ts_tablet_manager.cc:1397] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.006s
I20260809 04:54:14.673468 26082 raft_consensus.cc:348] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } }
I20260809 04:54:14.674839 26082 raft_consensus.cc:729] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303a5bf262f24e038319178ec5f34cdc, State: Initialized, Role: FOLLOWER
I20260809 04:54:14.675720 26082 consensus_queue.cc:260] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } }
I20260809 04:54:14.677340 26082 ts_tablet_manager.cc:1428] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260809 04:54:14.681166 25991 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:38963
I20260809 04:54:14.681198 26244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:38963 every 8 connection(s)
I20260809 04:54:14.682380 25991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:54:14.683506 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 25991
I20260809 04:54:14.684284 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:43375
--local_ip_for_outbound_sockets=127.31.93.2
--tserver_master_addrs=127.31.93.62:36055
--webserver_port=41231
--webserver_interface=127.31.93.2
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:54:14.687258 26250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:14.685203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:36135)
0809 04:54:14.686096 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:14.686371 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":381,"threads_started":1}
W20260809 04:54:14.687978 26246 heartbeater.cc:644] Failed to heartbeat to 127.31.93.62:36055 (0 consecutive failures): Network error: Failed to ping master at 127.31.93.62:36055: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
I20260809 04:54:14.688828 26250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:14.688490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:52761)
0809 04:54:14.688645 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:14.688719 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 04:54:14.689707 26250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:14.689357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:40171)
0809 04:54:14.689500 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:14.689580 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 04:54:14.689966 26246 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 04:54:15.027794 26252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:15.028146 26252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:15.028306 26252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:15.035493 26252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:15.035595 26252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:15.035746 26252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:54:15.045089 26252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:43375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=41231
--tserver_master_addrs=127.31.93.62:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:15.046267 26252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:15.047605 26252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:15.059895 26263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:15.060024 26265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:15.060122 26259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:15.060784 26252 server_base.cc:1034] running on GCE node
I20260809 04:54:15.061362 26252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:15.062279 26252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:15.063539 26252 hybrid_clock.cc:648] HybridClock initialized: now 1786251255063488 us; error 40 us; skew 500 ppm
I20260809 04:54:15.063838 26252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:15 dist-test-slave-4tdj krb5kdc[24236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:15.066869 26252 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:54:15.068176 26252 webserver.cc:460] Webserver started at http://127.31.93.2:41231/ using document root <none> and password file <none>
I20260809 04:54:15.068770 26252 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:15.068871 26252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:15.073978 26252 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260809 04:54:15.076687 26276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:15.077941 26252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 04:54:15.078088 26252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "372ec9e2a94043b6afa7c582c714a8aa"
format_stamp: "Formatted at 2026-08-09 04:53:29 on dist-test-slave-4tdj"
server_key: "428837f2e2c4b90fd770544fdb7a42e8"
server_key_iv: "e10973d94c39ee77840e850896749500"
server_key_version: "encryptionkey@0"
I20260809 04:54:15.079059 26252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:15.107920 26252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:15.111642 26252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:15.111887 26252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:15.112800 26252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:15.114078 26252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:15.114164 26252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:15.114267 26252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:15.114367 26252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 04:54:15.143289 26252 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:43375
I20260809 04:54:15.143352 26454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:43375 every 8 connection(s)
I20260809 04:54:15.144656 26252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:54:15.149199 26458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:15.147443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:56035)
0809 04:54:15.148142 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:15.148352 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":272,"threads_started":1}
W20260809 04:54:15.149868 26455 heartbeater.cc:644] Failed to heartbeat to 127.31.93.62:36055 (0 consecutive failures): Network error: Failed to ping master at 127.31.93.62:36055: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
I20260809 04:54:15.150622 26458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:15.150277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:48991)
0809 04:54:15.150435 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:15.150510 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260809 04:54:15.151378 26458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:15.151079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:57339)
0809 04:54:15.151196 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:15.151270 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 04:54:15.151616 26455 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 04:54:15.154083 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 26252
I20260809 04:54:15.154695 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:34407
--local_ip_for_outbound_sockets=127.31.93.3
--tserver_master_addrs=127.31.93.62:36055
--webserver_port=45215
--webserver_interface=127.31.93.3
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:38719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:15.470254 26459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:15.470541 26459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:15.470661 26459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:15.477474 26459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:15.477586 26459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:15.477736 26459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:54:15.486748 26459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:34407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=45215
--tserver_master_addrs=127.31.93.62:36055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:15.487887 26459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:15.489080 26459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:15.500907 26471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:15.501004 26469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:15.501044 26476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:15.501678 26459 server_base.cc:1034] running on GCE node
I20260809 04:54:15.502215 26459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:15.503063 26459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:15.504258 26459 hybrid_clock.cc:648] HybridClock initialized: now 1786251255504217 us; error 32 us; skew 500 ppm
I20260809 04:54:15.504525 26459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:15 dist-test-slave-4tdj krb5kdc[24236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:15.507441 26459 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:54:15.508680 26459 webserver.cc:460] Webserver started at http://127.31.93.3:45215/ using document root <none> and password file <none>
I20260809 04:54:15.509248 26459 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:15.509342 26459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:15.514146 26459 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:54:15.516587 26488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:15.517746 26459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:54:15.517886 26459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "34ba07b555464b4eb9783460041cd7db"
format_stamp: "Formatted at 2026-08-09 04:53:30 on dist-test-slave-4tdj"
server_key: "4619ee9cf0546af9dd304c76aec32437"
server_key_iv: "2ca18714654e6376cd4d0be6f9ffaba1"
server_key_version: "encryptionkey@0"
I20260809 04:54:15.518862 26459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:15.535413 26459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:15.538740 26459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:15.538960 26459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:15.539824 26459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:15.540992 26459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:15.541093 26459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:15.541198 26459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:15.541265 26459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:15.570667 26459 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:34407
I20260809 04:54:15.570737 26683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:34407 every 8 connection(s)
I20260809 04:54:15.571971 26459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:54:15.576354 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:15.574581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:40589)
0809 04:54:15.575259 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:15.575450 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":266,"threads_started":1}
W20260809 04:54:15.576936 26684 heartbeater.cc:644] Failed to heartbeat to 127.31.93.62:36055 (0 consecutive failures): Network error: Failed to ping master at 127.31.93.62:36055: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
I20260809 04:54:15.577620 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:15.577334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:47009)
0809 04:54:15.577446 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:15.577523 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 04:54:15.578274 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:15.578016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:35359)
0809 04:54:15.578112 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:15.578177 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 04:54:15.578486 26684 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 04:54:15.579075 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 26459
I20260809 04:54:15.691102 26690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:15.690403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:59259)
0809 04:54:15.690816 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:15.690899 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":194,"threads_started":1}
I20260809 04:54:15.933109 26693 raft_consensus.cc:482] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:54:15.933396 26693 raft_consensus.cc:504] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } }
I20260809 04:54:15.935024 26693 leader_election.cc:288] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375), 34ba07b555464b4eb9783460041cd7db (127.31.93.3:34407)
Aug 09 04:54:15 dist-test-slave-4tdj krb5kdc[24236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251254, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
Aug 09 04:54:15 dist-test-slave-4tdj krb5kdc[24236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251254, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:54:15.956285 26699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:15.936077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:38041 (local address 127.31.93.3:34407)
0809 04:54:15.938178 (+ 2101us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:15.938190 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:15.938229 (+ 39us) server_negotiation.cc:407] Connection header received
0809 04:54:15.939705 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:15.939714 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:15.940066 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:15.940350 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:15.942250 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:15.943443 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:15.945042 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:15.945658 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:15.945811 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:15.950216 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:15.950222 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:15.950232 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:15.950269 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:15.952821 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:15.953558 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:15.953566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:15.953573 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:15.953718 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:15.954151 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:15.954156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:15.954159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:15.954635 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:15.954782 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:15.955556 (+ 774us) server_negotiation.cc:299] Negotiation successful
0809 04:54:15.955877 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1946,"thread_start_us":1793,"threads_started":1}
I20260809 04:54:15.956305 26694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:15.935589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:50839 (local address 127.31.93.2:43375)
0809 04:54:15.938321 (+ 2732us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:15.938335 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:15.938371 (+ 36us) server_negotiation.cc:407] Connection header received
0809 04:54:15.939778 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:15.939786 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:15.940105 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:15.940403 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:15.942256 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:15.943438 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:15.945040 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:15.945667 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:15.945823 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:15.949961 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:15.949971 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:15.949987 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:15.950028 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:15.952815 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:15.953574 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:15.953580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:15.953585 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:15.953735 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:15.954156 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:15.954159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:15.954161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:15.954652 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:15.954816 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:15.955574 (+ 758us) server_negotiation.cc:299] Negotiation successful
0809 04:54:15.955931 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2560,"thread_start_us":2454,"threads_started":1}
I20260809 04:54:15.956362 26690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:15.935323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:43375 (local address 127.31.93.1:50839)
0809 04:54:15.935531 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:15.935581 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:15.936199 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:15.940529 (+ 4330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:15.940539 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:15.941327 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:15.942069 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:15.942089 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:15.943626 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:15.943629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:15.944910 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:15.944922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:15.945895 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:15.945903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:15.946140 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:15.946940 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:54:15.946964 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:15.949675 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:15.952947 (+ 3272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:15.952956 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:15.952970 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:15.953445 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:15.953849 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:15.953853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:15.953855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:15.954051 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:15.954786 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:15.954790 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:15.955100 (+ 310us) client_negotiation.cc:769] Sending connection context
0809 04:54:15.955330 (+ 230us) client_negotiation.cc:240] Negotiation successful
0809 04:54:15.955995 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":349,"spinlock_wait_cycles":27776}
I20260809 04:54:15.956310 26698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:15.935674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:34407 (local address 127.31.93.1:38041)
0809 04:54:15.937515 (+ 1841us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:15.937534 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:15.937825 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:15.940528 (+ 2703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:15.940539 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:15.941327 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:15.942073 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:15.942089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:15.943621 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:15.943624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:15.944911 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:15.944923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:15.945862 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:15.945873 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:15.946139 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:15.946926 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:54:15.946948 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:15.949992 (+ 3044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:15.952947 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:15.952958 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:15.952971 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:15.953437 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:15.953837 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:15.953842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:15.953846 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:15.954044 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:15.954775 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:15.954783 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:15.955098 (+ 315us) client_negotiation.cc:769] Sending connection context
0809 04:54:15.955331 (+ 233us) client_negotiation.cc:240] Negotiation successful
0809 04:54:15.955849 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1705,"mutex_wait_us":194,"spinlock_wait_cycles":55296,"thread_start_us":1519,"threads_started":1}
W20260809 04:54:15.959209 26077 leader_election.cc:341] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375): Not found: Tablet not found: 5e58313bc4894e4d96c124ff753aefa4
W20260809 04:54:15.959489 26077 leader_election.cc:341] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 34ba07b555464b4eb9783460041cd7db (127.31.93.3:34407): Not found: Tablet not found: 5e58313bc4894e4d96c124ff753aefa4
I20260809 04:54:15.959647 26077 leader_election.cc:302] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 303a5bf262f24e038319178ec5f34cdc; no voters: 34ba07b555464b4eb9783460041cd7db, 372ec9e2a94043b6afa7c582c714a8aa
I20260809 04:54:15.959986 26693 raft_consensus.cc:2726] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:54:16.152912 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:16.152066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:53119)
0809 04:54:16.152527 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:16.152659 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":228,"threads_started":1}
I20260809 04:54:16.579663 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:16.578961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:51715)
0809 04:54:16.579356 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:16.579455 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":199,"threads_started":1}
I20260809 04:54:16.692483 26703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:16.691867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:50473)
0809 04:54:16.692256 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:16.692333 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":195,"threads_started":1}
I20260809 04:54:17.154441 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:17.153752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:45885)
0809 04:54:17.154169 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:17.154250 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":231,"threads_started":1}
I20260809 04:54:17.581081 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:17.580422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:49579)
0809 04:54:17.580824 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:17.580898 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":207,"threads_started":1}
I20260809 04:54:17.646726 26709 raft_consensus.cc:482] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:54:17.646919 26709 raft_consensus.cc:504] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } }
I20260809 04:54:17.647594 26709 leader_election.cc:288] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375), 34ba07b555464b4eb9783460041cd7db (127.31.93.3:34407)
W20260809 04:54:17.648454 26077 leader_election.cc:341] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375): Not found: Tablet not found: 5e58313bc4894e4d96c124ff753aefa4
W20260809 04:54:17.648685 26077 leader_election.cc:341] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 34ba07b555464b4eb9783460041cd7db (127.31.93.3:34407): Not found: Tablet not found: 5e58313bc4894e4d96c124ff753aefa4
I20260809 04:54:17.648774 26077 leader_election.cc:302] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 303a5bf262f24e038319178ec5f34cdc; no voters: 34ba07b555464b4eb9783460041cd7db, 372ec9e2a94043b6afa7c582c714a8aa
I20260809 04:54:17.649012 26709 raft_consensus.cc:2726] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:54:17.693965 26710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:17.693218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:48191)
0809 04:54:17.693658 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:17.693742 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":212,"threads_started":1}
I20260809 04:54:18.155977 26712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:18.155264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:57259)
0809 04:54:18.155724 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:18.155807 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":223,"threads_started":1}
I20260809 04:54:18.582506 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:18.581796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:43471)
0809 04:54:18.582225 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:18.582338 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":228,"threads_started":1}
I20260809 04:54:18.695420 26716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:18.694753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:50179)
0809 04:54:18.695166 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:18.695269 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":211,"threads_started":1}
I20260809 04:54:19.157567 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:19.156805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:35933)
0809 04:54:19.157295 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:19.157378 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":250,"threads_started":1}
I20260809 04:54:19.584043 26718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:19.583346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:58633)
0809 04:54:19.583769 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:19.583855 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":193,"threads_started":1}
I20260809 04:54:19.697063 26719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:19.696311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:57145)
0809 04:54:19.696780 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:19.696875 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":224,"threads_started":1}
I20260809 04:54:20.159147 26720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:20.158447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:39637)
0809 04:54:20.158910 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:20.158993 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":238,"threads_started":1}
I20260809 04:54:20.552645 26721 raft_consensus.cc:482] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:54:20.552830 26721 raft_consensus.cc:504] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } }
I20260809 04:54:20.553390 26721 leader_election.cc:288] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375), 34ba07b555464b4eb9783460041cd7db (127.31.93.3:34407)
W20260809 04:54:20.554412 26077 leader_election.cc:341] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 34ba07b555464b4eb9783460041cd7db (127.31.93.3:34407): Not found: Tablet not found: 5e58313bc4894e4d96c124ff753aefa4
W20260809 04:54:20.554620 26077 leader_election.cc:341] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375): Not found: Tablet not found: 5e58313bc4894e4d96c124ff753aefa4
I20260809 04:54:20.554740 26077 leader_election.cc:302] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 303a5bf262f24e038319178ec5f34cdc; no voters: 34ba07b555464b4eb9783460041cd7db, 372ec9e2a94043b6afa7c582c714a8aa
I20260809 04:54:20.554991 26721 raft_consensus.cc:2726] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:54:20.585522 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:20.584829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:50557)
0809 04:54:20.585294 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:20.585377 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":250,"threads_started":1}
I20260809 04:54:20.698629 26724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:20.697837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:45641)
0809 04:54:20.698316 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:20.698424 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260809 04:54:21.160624 26726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:21.159927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:48453)
0809 04:54:21.160376 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:21.160454 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":246,"threads_started":1}
I20260809 04:54:21.587030 26727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:21.586268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:56681)
0809 04:54:21.586783 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:21.586861 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":251,"threads_started":1}
I20260809 04:54:21.700131 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:21.699379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:56261)
0809 04:54:21.699833 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:21.699923 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":228,"threads_started":1}
I20260809 04:54:22.162382 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:22.161494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:60597)
0809 04:54:22.162078 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:22.162173 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":278,"threads_started":1}
I20260809 04:54:22.588663 26732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:22.587856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:46801)
0809 04:54:22.588360 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:22.588444 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":224,"threads_started":1}
I20260809 04:54:22.701560 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:22.700881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:49329)
0809 04:54:22.701310 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:22.701387 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":207,"threads_started":1}
I20260809 04:54:22.816157 26735 raft_consensus.cc:482] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:54:22.816345 26735 raft_consensus.cc:504] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } }
I20260809 04:54:22.816977 26735 leader_election.cc:288] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375), 34ba07b555464b4eb9783460041cd7db (127.31.93.3:34407)
W20260809 04:54:22.817821 26077 leader_election.cc:341] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375): Not found: Tablet not found: 5e58313bc4894e4d96c124ff753aefa4
W20260809 04:54:22.818042 26077 leader_election.cc:341] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 34ba07b555464b4eb9783460041cd7db (127.31.93.3:34407): Not found: Tablet not found: 5e58313bc4894e4d96c124ff753aefa4
I20260809 04:54:22.818131 26077 leader_election.cc:302] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 303a5bf262f24e038319178ec5f34cdc; no voters: 34ba07b555464b4eb9783460041cd7db, 372ec9e2a94043b6afa7c582c714a8aa
I20260809 04:54:22.818392 26735 raft_consensus.cc:2726] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:54:23.163888 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:23.163219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:52249)
0809 04:54:23.163624 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:23.163704 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":208,"threads_started":1}
I20260809 04:54:23.590035 26737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:23.589407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:55561)
0809 04:54:23.589794 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:23.589869 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":197,"threads_started":1}
I20260809 04:54:23.703058 26738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:23.702386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:40351)
0809 04:54:23.702796 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:23.702880 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":213,"threads_started":1}
I20260809 04:54:24.165359 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:24.164665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:60859)
0809 04:54:24.165115 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:24.165195 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":209,"threads_started":1}
I20260809 04:54:24.591544 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:24.590847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:52761)
0809 04:54:24.591295 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:24.591396 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":216,"threads_started":1}
I20260809 04:54:24.704555 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:24.703906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:58715)
0809 04:54:24.704316 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:24.704401 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":223,"threads_started":1}
I20260809 04:54:25.166931 26743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:25.166187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:59867)
0809 04:54:25.166650 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:25.166737 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":248,"threads_started":1}
I20260809 04:54:25.189707 26744 raft_consensus.cc:482] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:54:25.189920 26744 raft_consensus.cc:504] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372ec9e2a94043b6afa7c582c714a8aa" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43375 } } peers { permanent_uuid: "34ba07b555464b4eb9783460041cd7db" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34407 } } peers { permanent_uuid: "303a5bf262f24e038319178ec5f34cdc" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 38963 } }
I20260809 04:54:25.190584 26744 leader_election.cc:288] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375), 34ba07b555464b4eb9783460041cd7db (127.31.93.3:34407)
W20260809 04:54:25.191442 26077 leader_election.cc:341] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 372ec9e2a94043b6afa7c582c714a8aa (127.31.93.2:43375): Not found: Tablet not found: 5e58313bc4894e4d96c124ff753aefa4
W20260809 04:54:25.191696 26077 leader_election.cc:341] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 34ba07b555464b4eb9783460041cd7db (127.31.93.3:34407): Not found: Tablet not found: 5e58313bc4894e4d96c124ff753aefa4
I20260809 04:54:25.191807 26077 leader_election.cc:302] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 303a5bf262f24e038319178ec5f34cdc; no voters: 34ba07b555464b4eb9783460041cd7db, 372ec9e2a94043b6afa7c582c714a8aa
I20260809 04:54:25.192031 26744 raft_consensus.cc:2726] T 5e58313bc4894e4d96c124ff753aefa4 P 303a5bf262f24e038319178ec5f34cdc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 04:54:25.593123 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:25.592321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:52235)
0809 04:54:25.592793 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:25.592915 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":245,"threads_started":1}
I20260809 04:54:25.705976 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:25.705325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:50803)
0809 04:54:25.705718 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:25.705815 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":193,"threads_started":1}
I20260809 04:54:26.168478 26747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:26.167755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:35853)
0809 04:54:26.168219 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:26.168313 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":221,"threads_started":1}
I20260809 04:54:26.594735 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:26.593966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:47005)
0809 04:54:26.594488 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:26.594585 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":258,"threads_started":1}
I20260809 04:54:26.707392 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:26.706750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:59361)
0809 04:54:26.707152 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:26.707234 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":212,"threads_started":1}
I20260809 04:54:27.169966 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:27.169294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:34985)
0809 04:54:27.169713 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:27.169789 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":220,"threads_started":1}
I20260809 04:54:27.596385 26754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:27.595627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:55601)
0809 04:54:27.596150 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:27.596226 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":322,"threads_started":1}
I20260809 04:54:27.708812 26755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:27.708157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:40587)
0809 04:54:27.708567 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:27.708654 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":201,"threads_started":1}
I20260809 04:54:28.171465 26757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:28.170778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:46087)
0809 04:54:28.171201 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:28.171289 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":213,"threads_started":1}
I20260809 04:54:28.597992 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:28.597172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:37971)
0809 04:54:28.597703 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:28.597794 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":260,"threads_started":1}
I20260809 04:54:28.710268 26761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:28.709515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:59419)
0809 04:54:28.709976 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:28.710064 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":237,"threads_started":1}
I20260809 04:54:29.172962 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:29.172244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:37511)
0809 04:54:29.172682 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:29.172764 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":224,"threads_started":1}
I20260809 04:54:29.599503 26764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:29.598758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:45479)
0809 04:54:29.599224 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:29.599304 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":245,"threads_started":1}
I20260809 04:54:29.712003 26767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:29.711278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.1:57525)
0809 04:54:29.711729 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:29.711817 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":223,"threads_started":1}
I20260809 04:54:30.174763 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:30.174013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.2:48607)
0809 04:54:30.174500 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:30.174593 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":259,"threads_started":1}
I20260809 04:54:30.584285 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 25991
I20260809 04:54:30.601116 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:30.600402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:36055 (local address 127.31.93.3:43193)
0809 04:54:30.600861 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:30.600970 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.62:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":245,"threads_started":1}
I20260809 04:54:30.602109 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:30.585347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.31.93.1:38963)
0809 04:54:30.585880 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:30.585887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:30.585917 (+ 30us) server_negotiation.cc:407] Connection header received
0809 04:54:30.585983 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:30.585990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:30.586203 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:30.586398 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:30.588057 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:30.589042 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:30.590193 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:30.590784 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:30.590877 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:30.596689 (+ 5812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:30.596700 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:30.596714 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:30.596745 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:30.599409 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:30.600067 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:30.600075 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:30.600077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:30.600215 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:30.600590 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:30.600594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:30.600597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:30.600938 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:30.601246 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:30.601561 (+ 315us) server_negotiation.cc:299] Negotiation successful
0809 04:54:30.601818 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":245,"threads_started":1}
I20260809 04:54:30.603322 26237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:30.692785 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 25991
I20260809 04:54:30.718994 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 26252
I20260809 04:54:30.731521 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:30.719735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34708 (local address 127.31.93.2:43375)
0809 04:54:30.720172 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:30.720176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:30.720193 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:54:30.720248 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:30.720251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:30.720312 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:30.720401 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:30.721648 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:30.722449 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:30.723757 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:30.724173 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:30.724259 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:30.726901 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:30.726906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:30.726909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:30.726941 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:30.729362 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:30.729938 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:30.729941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:30.729943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:30.730071 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:30.730376 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:30.730379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:30.730381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:30.730638 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:30.730822 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:30.731086 (+ 264us) server_negotiation.cc:299] Negotiation successful
0809 04:54:30.731245 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":222,"threads_started":1}
I20260809 04:54:30.732180 26443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:30.813824 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 26252
I20260809 04:54:30.837505 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 26459
I20260809 04:54:30.849849 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:30.838244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.31.93.3:34407)
0809 04:54:30.838818 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:30.838822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:30.838840 (+ 18us) server_negotiation.cc:407] Connection header received
0809 04:54:30.838893 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:30.838897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:30.838967 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:30.839057 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:30.840278 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:30.841047 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:30.842191 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:30.842655 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:30.842747 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:30.845140 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:30.845145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:30.845148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:30.845180 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:30.847674 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:30.848276 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:30.848281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:30.848284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:30.848445 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:30.848749 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:30.848752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:30.848754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:30.848994 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:30.849221 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:30.849403 (+ 182us) server_negotiation.cc:299] Negotiation successful
0809 04:54:30.849562 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":314,"threads_started":1}
I20260809 04:54:30.850562 26678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:30.931586 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 26459
2026-08-09T04:54:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64181 ms)
[----------] 30 tests from SecurityITest (292699 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:54:31 dist-test-slave-4tdj krb5kdc[26785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:54:31 dist-test-slave-4tdj krb5kdc[26785](info): set up 2 sockets
Aug 09 04:54:31 dist-test-slave-4tdj krb5kdc[26785](info): commencing operation
krb5kdc: starting...
W20260809 04:54:32.147338 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:54:32 dist-test-slave-4tdj krb5kdc[26785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:54:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:54:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:54:32.395488 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:40221
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:40221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:32.714485 26806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:32.714756 26806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:32.714852 26806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:32.721738 26806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:54:32.721829 26806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:32.721884 26806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:32.721946 26806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:54:32.721988 26806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:54:32.731215 26806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:40221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:40221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:32.732342 26806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:32.733544 26806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:32.744998 26812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:32.745133 26814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:32.745142 26818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:32.745931 26806 server_base.cc:1034] running on GCE node
I20260809 04:54:32.746768 26806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:32.748073 26806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:32.749300 26806 hybrid_clock.cc:648] HybridClock initialized: now 1786251272749251 us; error 30 us; skew 500 ppm
I20260809 04:54:32.749635 26806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:32 dist-test-slave-4tdj krb5kdc[26785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251272, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:32.752486 26806 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:54:32.753867 26806 webserver.cc:460] Webserver started at http://127.31.93.62:38899/ using document root <none> and password file <none>
I20260809 04:54:32.754534 26806 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:32.754655 26806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:32.754926 26806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:32.757685 26806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "77150ed2ad0a4c5b95092b80aed33f2e"
format_stamp: "Formatted at 2026-08-09 04:54:32 on dist-test-slave-4tdj"
server_key: "f3fa2881df2e9952f5bcaec2c2eaaf76"
server_key_iv: "90315bde9576583f0dfa7dc6cfc7ab35"
server_key_version: "encryptionkey@0"
I20260809 04:54:32.758432 26806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "77150ed2ad0a4c5b95092b80aed33f2e"
format_stamp: "Formatted at 2026-08-09 04:54:32 on dist-test-slave-4tdj"
server_key: "f3fa2881df2e9952f5bcaec2c2eaaf76"
server_key_iv: "90315bde9576583f0dfa7dc6cfc7ab35"
server_key_version: "encryptionkey@0"
I20260809 04:54:32.763218 26806 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:54:32.766539 26829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:32.767799 26806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:54:32.767951 26806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "77150ed2ad0a4c5b95092b80aed33f2e"
format_stamp: "Formatted at 2026-08-09 04:54:32 on dist-test-slave-4tdj"
server_key: "f3fa2881df2e9952f5bcaec2c2eaaf76"
server_key_iv: "90315bde9576583f0dfa7dc6cfc7ab35"
server_key_version: "encryptionkey@0"
I20260809 04:54:32.768115 26806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:32.786587 26806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:32.790032 26806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:32.790270 26806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:32.807186 26806 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:40221
I20260809 04:54:32.807240 26913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:40221 every 8 connection(s)
I20260809 04:54:32.808413 26806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:54:32.811877 26914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:32.816345 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 26806
I20260809 04:54:32.816577 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:54:32.817440 32116 external_mini_cluster.cc:1428] Setting key d9d002abf504b378df9684e8e8c0855c
I20260809 04:54:32.823576 26914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e: Bootstrap starting.
I20260809 04:54:32.827068 26914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:32.828233 26914 log.cc:824] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:32.830891 26914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e: No bootstrap required, opened a new log
Aug 09 04:54:32 dist-test-slave-4tdj krb5kdc[26785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:54:32.839627 26914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77150ed2ad0a4c5b95092b80aed33f2e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 40221 } }
I20260809 04:54:32.839977 26914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:32.840077 26914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77150ed2ad0a4c5b95092b80aed33f2e, State: Initialized, Role: FOLLOWER
I20260809 04:54:32.840166 26921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:32.819423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.31.93.62:40221)
0809 04:54:32.820159 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:32.820171 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:32.820212 (+ 41us) server_negotiation.cc:407] Connection header received
0809 04:54:32.822286 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:32.822344 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:32.822694 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:32.822982 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:32.824796 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:32.826096 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:32.827914 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:32.828583 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:32.828771 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:32.833154 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:32.833161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:32.833174 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:32.833218 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:32.836264 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:32.837040 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:32.837053 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:32.837059 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:32.837215 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:32.837587 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:32.837590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:32.837592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:32.838175 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:32.838464 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:32.838815 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 04:54:32.839128 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":343,"threads_started":1}
I20260809 04:54:32.840659 26914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77150ed2ad0a4c5b95092b80aed33f2e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 40221 } }
I20260809 04:54:32.840902 26914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:54:32.841033 26914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:54:32.841190 26914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:32.844019 26914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77150ed2ad0a4c5b95092b80aed33f2e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 40221 } }
I20260809 04:54:32.844481 26914 leader_election.cc:302] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77150ed2ad0a4c5b95092b80aed33f2e; no voters:
I20260809 04:54:32.845023 26914 leader_election.cc:288] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:54:32.845144 26923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:54:32.845793 26923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [term 1 LEADER]: Becoming Leader. State: Replica: 77150ed2ad0a4c5b95092b80aed33f2e, State: Running, Role: LEADER
I20260809 04:54:32.846310 26923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77150ed2ad0a4c5b95092b80aed33f2e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 40221 } }
I20260809 04:54:32.847072 26914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:54:32.849006 26924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77150ed2ad0a4c5b95092b80aed33f2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77150ed2ad0a4c5b95092b80aed33f2e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 40221 } } }
I20260809 04:54:32.848953 26925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77150ed2ad0a4c5b95092b80aed33f2e. Latest consensus state: current_term: 1 leader_uuid: "77150ed2ad0a4c5b95092b80aed33f2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77150ed2ad0a4c5b95092b80aed33f2e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 40221 } } }
I20260809 04:54:32.849414 26924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:32.849447 26925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:32.850364 26934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:54:32.855434 26934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:54:32.864092 26934 catalog_manager.cc:1347] Generated new cluster ID: c35db44c68544fa6ac33b80144c3e506
I20260809 04:54:32.864197 26934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:54:32.893576 26934 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:54:32.895027 26934 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:54:32.912408 26934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e: Generated new TSK 0
I20260809 04:54:32.913313 26934 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:54:32.977834 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:40221
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:33.307845 26959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:33.308112 26959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:33.308233 26959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:33.315069 26959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:33.315178 26959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:33.315326 26959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:54:33.324451 26959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:40221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:33.325659 26959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:33.326942 26959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:33.339272 26968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:33.339488 26967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:33.339272 26971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:33.340082 26959 server_base.cc:1034] running on GCE node
I20260809 04:54:33.340653 26959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:33.341536 26959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:33.342727 26959 hybrid_clock.cc:648] HybridClock initialized: now 1786251273342684 us; error 35 us; skew 500 ppm
I20260809 04:54:33.343010 26959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:33 dist-test-slave-4tdj krb5kdc[26785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251273, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:33.345867 26959 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:54:33.347234 26959 webserver.cc:460] Webserver started at http://127.31.93.1:33983/ using document root <none> and password file <none>
I20260809 04:54:33.347898 26959 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:33.348013 26959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:33.348345 26959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:33.351138 26959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "d111816812cc4eafbecc2dd521e5739a"
format_stamp: "Formatted at 2026-08-09 04:54:33 on dist-test-slave-4tdj"
server_key: "a2de2254737f46328331dcdc2a5acaec"
server_key_iv: "772ab0a21f12d6af20dcccb0cbbbcb98"
server_key_version: "encryptionkey@0"
I20260809 04:54:33.351819 26959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "d111816812cc4eafbecc2dd521e5739a"
format_stamp: "Formatted at 2026-08-09 04:54:33 on dist-test-slave-4tdj"
server_key: "a2de2254737f46328331dcdc2a5acaec"
server_key_iv: "772ab0a21f12d6af20dcccb0cbbbcb98"
server_key_version: "encryptionkey@0"
I20260809 04:54:33.356689 26959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 04:54:33.359890 26984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:33.361091 26959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:54:33.361227 26959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "d111816812cc4eafbecc2dd521e5739a"
format_stamp: "Formatted at 2026-08-09 04:54:33 on dist-test-slave-4tdj"
server_key: "a2de2254737f46328331dcdc2a5acaec"
server_key_iv: "772ab0a21f12d6af20dcccb0cbbbcb98"
server_key_version: "encryptionkey@0"
I20260809 04:54:33.361392 26959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:33.384701 26959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:33.388275 26959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:33.388535 26959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:33.389458 26959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:33.390712 26959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:33.390786 26959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:33.390902 26959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:33.390954 26959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:33.421191 26959 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:42927
I20260809 04:54:33.421277 27146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:42927 every 8 connection(s)
I20260809 04:54:33.422493 26959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:54:33.424474 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 26959
I20260809 04:54:33.424664 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:54:33.425482 32116 external_mini_cluster.cc:1428] Setting key 88f4087e59556c18a91bf6f60070e0c6
Aug 09 04:54:33 dist-test-slave-4tdj krb5kdc[26785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251273, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:54:33.442960 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:33.425072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:40289 (local address 127.31.93.62:40221)
0809 04:54:33.425575 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:33.425582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:33.426734 (+ 1152us) server_negotiation.cc:407] Connection header received
0809 04:54:33.428712 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:33.428718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:33.428797 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:33.428901 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:33.430417 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:33.431379 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:33.432937 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:33.433456 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:33.433588 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:33.437008 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:33.437014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:33.437019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:33.437056 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:33.439811 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:33.440683 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:33.440688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:33.440690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:33.440805 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:33.441267 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:33.441270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:33.441272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:33.441508 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:33.441624 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:33.442549 (+ 925us) server_negotiation.cc:299] Negotiation successful
0809 04:54:33.442710 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260809 04:54:33.444363 27153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:33.425655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:40221 (local address 127.31.93.1:40289)
0809 04:54:33.426538 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:33.426626 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:33.428430 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:33.429110 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:33.429118 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:33.429641 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:33.430226 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:33.430245 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:33.431522 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:33.431526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:33.432789 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:33.432802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:33.433658 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:33.433663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:33.433875 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:33.434578 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:54:33.434603 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:33.436789 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:33.439954 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:33.439964 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:33.439988 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:33.440561 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:33.440902 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:33.440907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:33.440912 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:33.441170 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:33.441639 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:33.441647 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:33.442068 (+ 421us) client_negotiation.cc:769] Sending connection context
0809 04:54:33.442452 (+ 384us) client_negotiation.cc:240] Negotiation successful
0809 04:54:33.442955 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":345,"threads_started":1}
I20260809 04:54:33.445957 27149 heartbeater.cc:347] Connected to a master server at 127.31.93.62:40221
I20260809 04:54:33.446331 27149 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:33.447321 27149 heartbeater.cc:511] Master 127.31.93.62:40221 requested a full tablet report, sending...
I20260809 04:54:33.449635 26858 ts_manager.cc:194] Registered new tserver with Master: d111816812cc4eafbecc2dd521e5739a (127.31.93.1:42927)
I20260809 04:54:33.451073 26858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:40289
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:54:33.487026 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:40221
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:33.801116 27156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:33.801358 27156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:33.801472 27156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:33.809314 27156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:33.809424 27156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:33.809626 27156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:54:33.824201 27156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:40221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:33.825603 27156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:33.826992 27156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:33.839133 27161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:33.839210 27162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:33.839162 27164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:33.839905 27156 server_base.cc:1034] running on GCE node
I20260809 04:54:33.840451 27156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:33.841281 27156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:33.842473 27156 hybrid_clock.cc:648] HybridClock initialized: now 1786251273842433 us; error 31 us; skew 500 ppm
I20260809 04:54:33.842756 27156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:33 dist-test-slave-4tdj krb5kdc[26785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251273, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:33.845639 27156 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:54:33.847002 27156 webserver.cc:460] Webserver started at http://127.31.93.2:37227/ using document root <none> and password file <none>
I20260809 04:54:33.847643 27156 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:33.847765 27156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:33.848052 27156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:33.850798 27156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "77d2a95d3ccf4cd18ace1add61964676"
format_stamp: "Formatted at 2026-08-09 04:54:33 on dist-test-slave-4tdj"
server_key: "a402915a8f092ecf0d31da89600c133b"
server_key_iv: "2cbad78ed68ab241cd69493f1eeb5c22"
server_key_version: "encryptionkey@0"
I20260809 04:54:33.851511 27156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "77d2a95d3ccf4cd18ace1add61964676"
format_stamp: "Formatted at 2026-08-09 04:54:33 on dist-test-slave-4tdj"
server_key: "a402915a8f092ecf0d31da89600c133b"
server_key_iv: "2cbad78ed68ab241cd69493f1eeb5c22"
server_key_version: "encryptionkey@0"
I20260809 04:54:33.856285 27156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:54:33.859520 27174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:33.860787 27156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:54:33.860927 27156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "77d2a95d3ccf4cd18ace1add61964676"
format_stamp: "Formatted at 2026-08-09 04:54:33 on dist-test-slave-4tdj"
server_key: "a402915a8f092ecf0d31da89600c133b"
server_key_iv: "2cbad78ed68ab241cd69493f1eeb5c22"
server_key_version: "encryptionkey@0"
I20260809 04:54:33.861099 27156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:33.877545 27156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:33.881317 27156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:33.881563 27156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:33.882457 27156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:33.883675 27156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:33.883752 27156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:33.883873 27156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:33.883927 27156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:33.914036 27156 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:40383
I20260809 04:54:33.914070 27347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:40383 every 8 connection(s)
I20260809 04:54:33.915270 27156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:54:33.919559 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 27156
I20260809 04:54:33.919751 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:54:33.920297 32116 external_mini_cluster.cc:1428] Setting key 8e28bb70a52304e5271bf0a34a263911
Aug 09 04:54:33 dist-test-slave-4tdj krb5kdc[26785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251273, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:54:33.934927 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:33.917694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:58921 (local address 127.31.93.62:40221)
0809 04:54:33.917874 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:33.917879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:33.919013 (+ 1134us) server_negotiation.cc:407] Connection header received
0809 04:54:33.921324 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:33.921330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:33.921412 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:33.921499 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:33.923215 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:33.923907 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:33.925409 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:33.925842 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:33.925966 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:33.929510 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:33.929515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:33.929518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:33.929551 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:33.931975 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:33.932768 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:33.932772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:33.932774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:33.932899 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:33.933320 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:33.933323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:33.933325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:33.933573 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:33.933679 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:33.934373 (+ 694us) server_negotiation.cc:299] Negotiation successful
0809 04:54:33.934603 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 04:54:33.935726 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:33.918150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:40221 (local address 127.31.93.2:58921)
0809 04:54:33.918865 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:33.918902 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:33.921059 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:33.921733 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:33.921741 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:33.922257 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:33.923034 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:33.923058 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:33.924062 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:33.924067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:33.925272 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:33.925284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:33.926068 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:33.926076 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:33.926285 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:33.926904 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:54:33.926930 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:33.929288 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:33.932113 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:33.932120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:33.932135 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:33.932643 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:33.932991 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:33.932995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:33.932998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:33.933232 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:33.933687 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:33.933695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:33.933989 (+ 294us) client_negotiation.cc:769] Sending connection context
0809 04:54:33.934211 (+ 222us) client_negotiation.cc:240] Negotiation successful
0809 04:54:33.934628 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":298,"threads_started":1}
I20260809 04:54:33.937018 27349 heartbeater.cc:347] Connected to a master server at 127.31.93.62:40221
I20260809 04:54:33.937312 27349 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:33.938153 27349 heartbeater.cc:511] Master 127.31.93.62:40221 requested a full tablet report, sending...
I20260809 04:54:33.939672 26858 ts_manager.cc:194] Registered new tserver with Master: 77d2a95d3ccf4cd18ace1add61964676 (127.31.93.2:40383)
I20260809 04:54:33.940498 26858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:58921
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:54:33.979779 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:40221
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:34.299732 27359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:34.299971 27359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:34.300067 27359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:34.306859 27359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:34.306950 27359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:34.307061 27359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:54:34.316063 27359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:40221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:34.317164 27359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:34.318395 27359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:34.330436 27367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:34.330394 27369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:34.330473 27371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:34.331296 27359 server_base.cc:1034] running on GCE node
I20260809 04:54:34.331851 27359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:34.332682 27359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:34.333879 27359 hybrid_clock.cc:648] HybridClock initialized: now 1786251274333841 us; error 26 us; skew 500 ppm
I20260809 04:54:34.334157 27359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:34 dist-test-slave-4tdj krb5kdc[26785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251274, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:34.337059 27359 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:54:34.338362 27359 webserver.cc:460] Webserver started at http://127.31.93.3:39917/ using document root <none> and password file <none>
I20260809 04:54:34.338985 27359 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:34.339102 27359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:34.339385 27359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:34.342175 27359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "799cf5455ad045de987de03f05c25371"
format_stamp: "Formatted at 2026-08-09 04:54:34 on dist-test-slave-4tdj"
server_key: "eb88e064bcf486af11dff36159fc2038"
server_key_iv: "78e2adeb021f60b1f1f8849c8afe50fb"
server_key_version: "encryptionkey@0"
I20260809 04:54:34.342953 27359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "799cf5455ad045de987de03f05c25371"
format_stamp: "Formatted at 2026-08-09 04:54:34 on dist-test-slave-4tdj"
server_key: "eb88e064bcf486af11dff36159fc2038"
server_key_iv: "78e2adeb021f60b1f1f8849c8afe50fb"
server_key_version: "encryptionkey@0"
I20260809 04:54:34.347677 27359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:54:34.350844 27381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:34.352015 27359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260809 04:54:34.352155 27359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "799cf5455ad045de987de03f05c25371"
format_stamp: "Formatted at 2026-08-09 04:54:34 on dist-test-slave-4tdj"
server_key: "eb88e064bcf486af11dff36159fc2038"
server_key_iv: "78e2adeb021f60b1f1f8849c8afe50fb"
server_key_version: "encryptionkey@0"
I20260809 04:54:34.352325 27359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:34.362728 27359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:34.366639 27359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:34.366879 27359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:34.367687 27359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:34.368847 27359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:34.368917 27359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:34.369028 27359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:34.369081 27359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:34.398767 27359 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:40839
I20260809 04:54:34.398826 27560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:40839 every 8 connection(s)
I20260809 04:54:34.399966 27359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:54:34.401839 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 27359
I20260809 04:54:34.401991 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:54:34.402585 32116 external_mini_cluster.cc:1428] Setting key c1a2ca4e96deac853bf5d94b73d60a12
Aug 09 04:54:34 dist-test-slave-4tdj krb5kdc[26785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251274, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:54:34.418900 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.402344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:34891 (local address 127.31.93.62:40221)
0809 04:54:34.402562 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:34.402568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:34.403764 (+ 1196us) server_negotiation.cc:407] Connection header received
0809 04:54:34.405950 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:34.405955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:34.406042 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:34.406129 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:34.407618 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.408349 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.409837 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.410242 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.410397 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:34.413565 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:34.413570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:34.413573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:34.413606 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:34.416073 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:34.416717 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:34.416722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:34.416725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:34.416862 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:34.417266 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:34.417270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:34.417274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:34.417530 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:34.417665 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:34.418337 (+ 672us) server_negotiation.cc:299] Negotiation successful
0809 04:54:34.418572 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 04:54:34.419729 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.402835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:40221 (local address 127.31.93.3:34891)
0809 04:54:34.403605 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:34.403656 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:34.405680 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:34.406399 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:34.406408 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:34.406904 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:34.407454 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:34.407471 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.408491 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.408497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:34.409695 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:34.409713 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.410493 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.410500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:34.410748 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:34.411343 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:54:34.411364 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:34.413370 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:34.416196 (+ 2826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:34.416205 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:34.416219 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:34.416605 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:34.416962 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:34.416967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:34.416971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:34.417157 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:34.417658 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:34.417665 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:34.417948 (+ 283us) client_negotiation.cc:769] Sending connection context
0809 04:54:34.418186 (+ 238us) client_negotiation.cc:240] Negotiation successful
0809 04:54:34.418679 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":306,"threads_started":1}
I20260809 04:54:34.420979 27561 heartbeater.cc:347] Connected to a master server at 127.31.93.62:40221
I20260809 04:54:34.421255 27561 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:34.421893 27561 heartbeater.cc:511] Master 127.31.93.62:40221 requested a full tablet report, sending...
I20260809 04:54:34.423255 26858 ts_manager.cc:194] Registered new tserver with Master: 799cf5455ad045de987de03f05c25371 (127.31.93.3:40839)
I20260809 04:54:34.424021 26858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:34891
I20260809 04:54:34.429502 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:54:34.445044 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.433863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.31.93.62:40221)
0809 04:54:34.434013 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:34.434017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:34.434508 (+ 491us) server_negotiation.cc:407] Connection header received
0809 04:54:34.434649 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:34.434653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:34.434721 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:34.434818 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:34.436099 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.436790 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.438020 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.438437 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.438524 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:34.440667 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:34.440672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:34.440676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:34.440709 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:34.442919 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:34.443531 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:34.443536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:34.443538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:34.443688 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:34.443990 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:34.443993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:34.443995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:34.444282 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:34.444458 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:34.444697 (+ 239us) server_negotiation.cc:299] Negotiation successful
0809 04:54:34.444842 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:54:34.448706 26858 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 04:54:34.454523 27149 heartbeater.cc:503] Master 127.31.93.62:40221 was elected leader, sending a full tablet report...
W20260809 04:54:34.455578 26858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:54:34.493541 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.468019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56934 (local address 127.31.93.3:40839)
0809 04:54:34.485274 (+ 17255us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:34.485283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:34.485320 (+ 37us) server_negotiation.cc:407] Connection header received
0809 04:54:34.485428 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:34.485436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:34.485685 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:34.485883 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:34.486631 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.488156 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.490789 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.491672 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.491805 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:34.492034 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:34.493038 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0809 04:54:34.493208 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17075,"thread_start_us":16888,"threads_started":1}
I20260809 04:54:34.493621 27578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.467055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:40839 (local address 127.0.0.1:56934)
0809 04:54:34.482916 (+ 15861us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:34.482958 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:34.483143 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:34.485901 (+ 2758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:34.485910 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:34.485955 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:34.486395 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:34.486408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.488282 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.488287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:34.490636 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:34.490651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.492428 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.492435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:34.492756 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:34.492820 (+ 64us) client_negotiation.cc:769] Sending connection context
0809 04:54:34.492958 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 04:54:34.493155 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15620,"mutex_wait_us":99,"thread_start_us":15484,"threads_started":1}
I20260809 04:54:34.495186 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.467502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:40383 (local address 127.0.0.1:38230)
0809 04:54:34.486589 (+ 19087us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:34.486608 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:34.486731 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:34.488241 (+ 1510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:34.488246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:34.488265 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:34.488803 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:34.488815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.490413 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.490418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:34.492737 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:34.492751 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.494413 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.494420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:34.494589 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:34.494612 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:54:34.494699 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 04:54:34.494826 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18940,"mutex_wait_us":79,"thread_start_us":20221,"threads_started":1}
I20260809 04:54:34.495211 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.468639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.31.93.2:40383)
0809 04:54:34.484736 (+ 16097us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:34.484745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:34.486851 (+ 2106us) server_negotiation.cc:407] Connection header received
0809 04:54:34.486972 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:34.486980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:34.487218 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:34.487399 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:34.489011 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.490218 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.492885 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.494191 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.494327 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:34.494503 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:34.494758 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 04:54:34.494908 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15926,"thread_start_us":15823,"threads_started":1}
I20260809 04:54:34.497537 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.467702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42927 (local address 127.0.0.1:59106)
0809 04:54:34.487992 (+ 20290us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:34.488011 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:34.488130 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:34.490240 (+ 2110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:34.490245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:34.490275 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:34.490917 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:34.490934 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.492774 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.492779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:34.494818 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:34.494833 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.496816 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.496823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:34.496976 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:34.497006 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 04:54:34.497083 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 04:54:34.497195 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20167,"mutex_wait_us":22,"thread_start_us":20590,"threads_started":1}
I20260809 04:54:34.498342 27585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.468079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.31.93.1:42927)
0809 04:54:34.489733 (+ 21654us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:34.489740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:34.489766 (+ 26us) server_negotiation.cc:407] Connection header received
0809 04:54:34.489851 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:34.489858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:34.490033 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:34.490245 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:34.491110 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.492604 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.495007 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.496343 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.496459 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:34.496669 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:34.497870 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0809 04:54:34.498020 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21493,"thread_start_us":21351,"threads_started":1}
I20260809 04:54:34.497759 27457 tablet_service.cc:1467] Processing CreateTablet for tablet 0f054b78228241c387db5c7ea277c10f (DEFAULT_TABLE table=test-table [id=adf3a749e61d43a7b79ef2b226399fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:34.498394 27258 tablet_service.cc:1467] Processing CreateTablet for tablet 0f054b78228241c387db5c7ea277c10f (DEFAULT_TABLE table=test-table [id=adf3a749e61d43a7b79ef2b226399fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:34.499367 27457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f054b78228241c387db5c7ea277c10f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:34.499511 27258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f054b78228241c387db5c7ea277c10f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:34.501724 27056 tablet_service.cc:1467] Processing CreateTablet for tablet 0f054b78228241c387db5c7ea277c10f (DEFAULT_TABLE table=test-table [id=adf3a749e61d43a7b79ef2b226399fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:34.503017 27056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f054b78228241c387db5c7ea277c10f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:34.510051 27589 tablet_bootstrap.cc:492] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371: Bootstrap starting.
I20260809 04:54:34.513247 27589 tablet_bootstrap.cc:654] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:34.514196 27589 log.cc:824] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:34.516561 27592 tablet_bootstrap.cc:492] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a: Bootstrap starting.
I20260809 04:54:34.516875 27589 tablet_bootstrap.cc:492] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371: No bootstrap required, opened a new log
I20260809 04:54:34.517194 27589 ts_tablet_manager.cc:1397] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260809 04:54:34.519269 27592 tablet_bootstrap.cc:654] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:34.520305 27592 log.cc:824] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:34.522557 27592 tablet_bootstrap.cc:492] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a: No bootstrap required, opened a new log
I20260809 04:54:34.522773 27592 ts_tablet_manager.cc:1397] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260809 04:54:34.522753 27590 tablet_bootstrap.cc:492] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676: Bootstrap starting.
I20260809 04:54:34.524849 27589 raft_consensus.cc:348] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "799cf5455ad045de987de03f05c25371" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40839 } } peers { permanent_uuid: "77d2a95d3ccf4cd18ace1add61964676" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40383 } } peers { permanent_uuid: "d111816812cc4eafbecc2dd521e5739a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42927 } }
I20260809 04:54:34.525421 27589 raft_consensus.cc:374] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:34.525530 27589 raft_consensus.cc:729] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 799cf5455ad045de987de03f05c25371, State: Initialized, Role: FOLLOWER
I20260809 04:54:34.526078 27590 tablet_bootstrap.cc:654] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:34.526141 27589 consensus_queue.cc:260] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "799cf5455ad045de987de03f05c25371" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40839 } } peers { permanent_uuid: "77d2a95d3ccf4cd18ace1add61964676" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40383 } } peers { permanent_uuid: "d111816812cc4eafbecc2dd521e5739a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42927 } }
I20260809 04:54:34.527227 27561 heartbeater.cc:503] Master 127.31.93.62:40221 was elected leader, sending a full tablet report...
I20260809 04:54:34.527288 27590 log.cc:824] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:34.529382 27589 ts_tablet_manager.cc:1428] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260809 04:54:34.531288 27592 raft_consensus.cc:348] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "799cf5455ad045de987de03f05c25371" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40839 } } peers { permanent_uuid: "77d2a95d3ccf4cd18ace1add61964676" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40383 } } peers { permanent_uuid: "d111816812cc4eafbecc2dd521e5739a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42927 } }
I20260809 04:54:34.531631 27592 raft_consensus.cc:374] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:34.531734 27592 raft_consensus.cc:729] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d111816812cc4eafbecc2dd521e5739a, State: Initialized, Role: FOLLOWER
I20260809 04:54:34.531998 27590 tablet_bootstrap.cc:492] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676: No bootstrap required, opened a new log
I20260809 04:54:34.532295 27590 ts_tablet_manager.cc:1397] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20260809 04:54:34.532351 27592 consensus_queue.cc:260] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "799cf5455ad045de987de03f05c25371" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40839 } } peers { permanent_uuid: "77d2a95d3ccf4cd18ace1add61964676" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40383 } } peers { permanent_uuid: "d111816812cc4eafbecc2dd521e5739a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42927 } }
I20260809 04:54:34.534132 27592 ts_tablet_manager.cc:1428] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a: Time spent starting tablet: real 0.011s user 0.002s sys 0.008s
I20260809 04:54:34.539733 27590 raft_consensus.cc:348] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "799cf5455ad045de987de03f05c25371" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40839 } } peers { permanent_uuid: "77d2a95d3ccf4cd18ace1add61964676" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40383 } } peers { permanent_uuid: "d111816812cc4eafbecc2dd521e5739a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42927 } }
I20260809 04:54:34.540091 27590 raft_consensus.cc:374] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:34.540184 27590 raft_consensus.cc:729] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77d2a95d3ccf4cd18ace1add61964676, State: Initialized, Role: FOLLOWER
I20260809 04:54:34.540642 27590 consensus_queue.cc:260] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "799cf5455ad045de987de03f05c25371" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40839 } } peers { permanent_uuid: "77d2a95d3ccf4cd18ace1add61964676" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40383 } } peers { permanent_uuid: "d111816812cc4eafbecc2dd521e5739a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42927 } }
I20260809 04:54:34.541615 27349 heartbeater.cc:503] Master 127.31.93.62:40221 was elected leader, sending a full tablet report...
I20260809 04:54:34.542157 27590 ts_tablet_manager.cc:1428] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
W20260809 04:54:34.651531 27563 tablet.cc:2369] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:54:34.654851 27604 raft_consensus.cc:482] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:54:34.655130 27604 raft_consensus.cc:504] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "799cf5455ad045de987de03f05c25371" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40839 } } peers { permanent_uuid: "77d2a95d3ccf4cd18ace1add61964676" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40383 } } peers { permanent_uuid: "d111816812cc4eafbecc2dd521e5739a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42927 } }
I20260809 04:54:34.656883 27604 leader_election.cc:288] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 799cf5455ad045de987de03f05c25371 (127.31.93.3:40839), d111816812cc4eafbecc2dd521e5739a (127.31.93.1:42927)
I20260809 04:54:34.663393 27585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.657509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:54797 (local address 127.31.93.1:42927)
0809 04:54:34.657727 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:34.657732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:34.658331 (+ 599us) server_negotiation.cc:407] Connection header received
0809 04:54:34.658506 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:34.658509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:34.658570 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:34.658642 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:34.659244 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.659970 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.661756 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.662565 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.662685 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:34.662783 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:34.663038 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 04:54:34.663166 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 04:54:34.663697 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.657184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:56561 (local address 127.31.93.3:40839)
0809 04:54:34.657428 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:34.657436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:34.658268 (+ 832us) server_negotiation.cc:407] Connection header received
0809 04:54:34.658504 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:34.658509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:34.658573 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:34.658656 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:34.659382 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.660129 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.661935 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.662778 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.662908 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:34.663027 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:34.663316 (+ 289us) server_negotiation.cc:299] Negotiation successful
0809 04:54:34.663441 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260809 04:54:34.663728 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.657310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42927 (local address 127.31.93.2:54797)
0809 04:54:34.658198 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:34.658212 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:34.658378 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:34.658669 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:34.658674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:34.658697 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:34.659082 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:34.659091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.660091 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.660096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:34.661604 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:34.661616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.662771 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.662777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:34.662896 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:34.662919 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:54:34.663009 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 04:54:34.663136 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":21,"spinlock_wait_cycles":29952,"thread_start_us":589,"threads_started":1}
I20260809 04:54:34.663861 27605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.657223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:40839 (local address 127.31.93.2:56561)
0809 04:54:34.658138 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:34.658166 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:34.658353 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:34.658684 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:34.658688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:34.658713 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:34.659236 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:34.659246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.660296 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.660300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:34.661772 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:34.661785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.663001 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.663008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:34.663166 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:34.663191 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 04:54:34.663287 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 04:54:34.663416 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":19,"thread_start_us":454,"threads_started":1}
I20260809 04:54:34.664866 27496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f054b78228241c387db5c7ea277c10f" candidate_uuid: "77d2a95d3ccf4cd18ace1add61964676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "799cf5455ad045de987de03f05c25371" is_pre_election: true
I20260809 04:54:34.664860 27089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f054b78228241c387db5c7ea277c10f" candidate_uuid: "77d2a95d3ccf4cd18ace1add61964676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d111816812cc4eafbecc2dd521e5739a" is_pre_election: true
I20260809 04:54:34.665383 27496 raft_consensus.cc:2454] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77d2a95d3ccf4cd18ace1add61964676 in term 0.
I20260809 04:54:34.665417 27089 raft_consensus.cc:2454] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77d2a95d3ccf4cd18ace1add61964676 in term 0.
I20260809 04:54:34.666059 27175 leader_election.cc:302] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77d2a95d3ccf4cd18ace1add61964676, 799cf5455ad045de987de03f05c25371; no voters:
I20260809 04:54:34.666429 27604 raft_consensus.cc:2781] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:54:34.666574 27604 raft_consensus.cc:482] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:54:34.666687 27604 raft_consensus.cc:3037] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [term 0 FOLLOWER]: Advancing to term 1
W20260809 04:54:34.667044 27354 tablet.cc:2369] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:54:34.669085 27604 raft_consensus.cc:504] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "799cf5455ad045de987de03f05c25371" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40839 } } peers { permanent_uuid: "77d2a95d3ccf4cd18ace1add61964676" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40383 } } peers { permanent_uuid: "d111816812cc4eafbecc2dd521e5739a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42927 } }
I20260809 04:54:34.669677 27604 leader_election.cc:288] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [CANDIDATE]: Term 1 election: Requested vote from peers 799cf5455ad045de987de03f05c25371 (127.31.93.3:40839), d111816812cc4eafbecc2dd521e5739a (127.31.93.1:42927)
I20260809 04:54:34.670034 27496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f054b78228241c387db5c7ea277c10f" candidate_uuid: "77d2a95d3ccf4cd18ace1add61964676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "799cf5455ad045de987de03f05c25371"
I20260809 04:54:34.670192 27089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f054b78228241c387db5c7ea277c10f" candidate_uuid: "77d2a95d3ccf4cd18ace1add61964676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d111816812cc4eafbecc2dd521e5739a"
I20260809 04:54:34.670230 27496 raft_consensus.cc:3037] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:34.670404 27089 raft_consensus.cc:3037] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:34.673357 27496 raft_consensus.cc:2454] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77d2a95d3ccf4cd18ace1add61964676 in term 1.
I20260809 04:54:34.673389 27089 raft_consensus.cc:2454] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77d2a95d3ccf4cd18ace1add61964676 in term 1.
I20260809 04:54:34.673729 27175 leader_election.cc:302] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77d2a95d3ccf4cd18ace1add61964676, 799cf5455ad045de987de03f05c25371; no voters:
I20260809 04:54:34.674026 27604 raft_consensus.cc:2781] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:54:34.674556 27604 raft_consensus.cc:686] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [term 1 LEADER]: Becoming Leader. State: Replica: 77d2a95d3ccf4cd18ace1add61964676, State: Running, Role: LEADER
W20260809 04:54:34.674679 27150 tablet.cc:2369] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:54:34.675103 27604 consensus_queue.cc:237] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "799cf5455ad045de987de03f05c25371" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40839 } } peers { permanent_uuid: "77d2a95d3ccf4cd18ace1add61964676" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40383 } } peers { permanent_uuid: "d111816812cc4eafbecc2dd521e5739a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42927 } }
I20260809 04:54:34.680212 26858 catalog_manager.cc:5495] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77d2a95d3ccf4cd18ace1add61964676 (127.31.93.2). New cstate: current_term: 1 leader_uuid: "77d2a95d3ccf4cd18ace1add61964676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "799cf5455ad045de987de03f05c25371" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77d2a95d3ccf4cd18ace1add61964676" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d111816812cc4eafbecc2dd521e5739a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42927 } health_report { overall_health: UNKNOWN } } }
I20260809 04:54:34.734419 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.729079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.31.93.2:40383)
0809 04:54:34.729294 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:34.729300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:34.729320 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:54:34.729407 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:34.729411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:34.729493 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:34.729624 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:54:34.730002 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.730796 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.732344 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.732739 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.732852 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:34.733343 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:54:34.733818 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:54:34.733949 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:34.734049 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 04:54:34.734134 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 04:54:34.742385 27089 raft_consensus.cc:1261] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a [term 1 FOLLOWER]: Refusing update from remote peer 77d2a95d3ccf4cd18ace1add61964676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:54:34.742383 27496 raft_consensus.cc:1261] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371 [term 1 FOLLOWER]: Refusing update from remote peer 77d2a95d3ccf4cd18ace1add61964676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:54:34.743444 27604 consensus_queue.cc:1035] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "799cf5455ad045de987de03f05c25371" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:54:34.743755 27607 consensus_queue.cc:1035] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d111816812cc4eafbecc2dd521e5739a" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:54:34.752174 27614 mvcc.cc:205] Tried to move back new op lower bound from 7316485221334278144 to 7316485221073641472. Current Snapshot: MvccSnapshot[applied={T|T < 7316485221334278144}]
I20260809 04:54:34.766736 26858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:54:34.767021 26858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:54:34.771579 26858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77150ed2ad0a4c5b95092b80aed33f2e: Sending DeleteTablet for 3 replicas of tablet 0f054b78228241c387db5c7ea277c10f
I20260809 04:54:34.772782 27258 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f054b78228241c387db5c7ea277c10f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:34 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:38230
I20260809 04:54:34.773964 27457 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f054b78228241c387db5c7ea277c10f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:34 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:56934
I20260809 04:54:34.774842 27627 tablet_replica.cc:330] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371: stopping tablet replica
I20260809 04:54:34.774850 27056 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f054b78228241c387db5c7ea277c10f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:34 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:59106
I20260809 04:54:34.775439 27627 raft_consensus.cc:2229] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:54:34.775949 27627 raft_consensus.cc:2258] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:34.777969 27627 ts_tablet_manager.cc:1905] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:54:34.781180 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 26959
I20260809 04:54:34.784981 27627 ts_tablet_manager.cc:1918] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:54:34.785200 27627 log.cc:1196] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/0f054b78228241c387db5c7ea277c10f
I20260809 04:54:34.785804 27627 ts_tablet_manager.cc:1939] T 0f054b78228241c387db5c7ea277c10f P 799cf5455ad045de987de03f05c25371: Deleting consensus metadata
I20260809 04:54:34.787539 26832 catalog_manager.cc:4841] TS 799cf5455ad045de987de03f05c25371 (127.31.93.3:40839): tablet 0f054b78228241c387db5c7ea277c10f (table test-table [id=adf3a749e61d43a7b79ef2b226399fe3]) successfully deleted
Aug 09 04:54:34 dist-test-slave-4tdj krb5kdc[26785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:54:34.805543 27628 tablet_replica.cc:330] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a: stopping tablet replica
I20260809 04:54:34.806610 27624 tablet_replica.cc:330] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676: stopping tablet replica
I20260809 04:54:34.807009 27624 raft_consensus.cc:2229] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:54:34.807787 27624 raft_consensus.cc:2258] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:34.810525 27628 raft_consensus.cc:2229] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:54:34.811452 27628 raft_consensus.cc:2258] T 0f054b78228241c387db5c7ea277c10f P d111816812cc4eafbecc2dd521e5739a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:34.811867 27624 ts_tablet_manager.cc:1905] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:54:34.815442 27585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.782991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.31.93.1:42927)
0809 04:54:34.783171 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:34.783176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:34.790743 (+ 7567us) server_negotiation.cc:407] Connection header received
0809 04:54:34.790916 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:34.790923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:34.791028 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:34.791123 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:34.793009 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.794360 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.796698 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.797415 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.797511 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:34.803323 (+ 5812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:34.803340 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:34.803359 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:34.803410 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:34.812170 (+ 8760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:34.812997 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:34.813005 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:34.813008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:34.813168 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:34.813633 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:34.813638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:34.813642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:34.814055 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:34.814330 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:34.814886 (+ 556us) server_negotiation.cc:299] Negotiation successful
0809 04:54:34.815096 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:54:34.816982 27141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:34.818991 27624 ts_tablet_manager.cc:1918] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:54:34.819164 27624 log.cc:1196] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/0f054b78228241c387db5c7ea277c10f
I20260809 04:54:34.819715 27624 ts_tablet_manager.cc:1939] T 0f054b78228241c387db5c7ea277c10f P 77d2a95d3ccf4cd18ace1add61964676: Deleting consensus metadata
I20260809 04:54:34.821388 26838 catalog_manager.cc:4841] TS 77d2a95d3ccf4cd18ace1add61964676 (127.31.93.2:40383): tablet 0f054b78228241c387db5c7ea277c10f (table test-table [id=adf3a749e61d43a7b79ef2b226399fe3]) successfully deleted
I20260809 04:54:34.946751 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 26959
W20260809 04:54:34.969731 26837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.93.1:42927 (error 108)
I20260809 04:54:34.969861 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 27156
I20260809 04:54:34.971892 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.971329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42927 (local address 127.0.0.1:59132)
0809 04:54:34.971582 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:34.971693 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260809 04:54:34.972155 26837 catalog_manager.cc:4584] TS d111816812cc4eafbecc2dd521e5739a (127.31.93.1:42927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f054b78228241c387db5c7ea277c10f: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42927: connect: Connection refused (error 111)
Aug 09 04:54:34 dist-test-slave-4tdj krb5kdc[26785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:54:34.985426 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:34.970856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.31.93.2:40383)
0809 04:54:34.971035 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:34.971040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:34.971060 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:54:34.971206 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:34.971210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:34.971298 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:34.971390 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:34.972839 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.973676 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.975024 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:34.975470 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:34.975589 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:34.979837 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:34.979848 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:34.979862 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:34.979900 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:34.982741 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:34.983552 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:34.983569 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:34.983571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:34.983709 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:34.984088 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:34.984091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:34.984093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:34.984420 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:34.984672 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:34.984990 (+ 318us) server_negotiation.cc:299] Negotiation successful
0809 04:54:34.985142 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:54:34.986135 27340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:35.025739 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:35.025176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42927 (local address 127.0.0.1:59134)
0809 04:54:35.025460 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:35.025573 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260809 04:54:35.026983 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:35.026539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42927 (local address 127.0.0.1:59140)
0809 04:54:35.026756 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:35.026846 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260809 04:54:35.087491 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:35.086959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42927 (local address 127.0.0.1:59142)
0809 04:54:35.087184 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:35.087296 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 04:54:35.088452 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:35.088110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42927 (local address 127.0.0.1:59152)
0809 04:54:35.088244 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:35.088327 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 04:54:35.113533 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 27156
I20260809 04:54:35.136394 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 27359
Aug 09 04:54:35 dist-test-slave-4tdj krb5kdc[26785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:54:35.150883 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:35.137125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56944 (local address 127.31.93.3:40839)
0809 04:54:35.137289 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:35.137293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:35.137356 (+ 63us) server_negotiation.cc:407] Connection header received
0809 04:54:35.137542 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:35.137546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:35.137627 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:35.137705 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:35.139068 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:35.139780 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:35.141012 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:35.141391 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:35.141483 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:35.145725 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:35.145739 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:35.145755 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:35.145786 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:35.148425 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:35.149080 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:35.149087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:35.149089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:35.149221 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:35.149622 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:35.149626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:35.149628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:35.149961 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:35.150196 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:35.150459 (+ 263us) server_negotiation.cc:299] Negotiation successful
0809 04:54:35.150635 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 04:54:35.151526 27553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:35.169108 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:35.168661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42927 (local address 127.0.0.1:59162)
0809 04:54:35.168854 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:35.168941 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 04:54:35.170087 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:35.169708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42927 (local address 127.0.0.1:59172)
0809 04:54:35.169897 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:35.169979 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260809 04:54:35.238438 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 27359
I20260809 04:54:35.256494 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 26806
I20260809 04:54:35.257637 26907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:35.365585 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:35.365100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42927 (local address 127.0.0.1:59188)
0809 04:54:35.365272 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:35.365363 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 04:54:35.365553 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 26806
2026-08-09T04:54:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4439 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-09T04:54:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:54:35Z Disabled control of system clock
I20260809 04:54:35.448781 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:44321
--webserver_interface=127.31.93.62
--webserver_port=0
--builtin_ntp_servers=127.31.93.20:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:44321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 04:54:35.765151 27646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:35.765436 27646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:35.765566 27646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:35.772538 27646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:54:35.772642 27646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:35.772707 27646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:35.772784 27646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:54:35.772838 27646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:54:35.781636 27646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:44321
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:44321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:35.782814 27646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:35.784487 27646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:35.795680 27651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:35.795823 27652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:35.795831 27656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:35.796705 27646 server_base.cc:1034] running on GCE node
I20260809 04:54:35.797484 27646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:35.798799 27646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:35.799988 27646 hybrid_clock.cc:648] HybridClock initialized: now 1786251275799936 us; error 48 us; skew 500 ppm
I20260809 04:54:35.800319 27646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:54:35.806025 27646 webserver.cc:460] Webserver started at http://127.31.93.62:42299/ using document root <none> and password file <none>
I20260809 04:54:35.806725 27646 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:35.806818 27646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:35.807107 27646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:35.809844 27646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "46a331b3af7a427eb33f11c415d99507"
format_stamp: "Formatted at 2026-08-09 04:54:35 on dist-test-slave-4tdj"
server_key: "36a75d93f227c44975b979c30f882c87"
server_key_iv: "a41788e54670968ddcbdcb7ac3ee120f"
server_key_version: "encryptionkey@0"
I20260809 04:54:35.810650 27646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "46a331b3af7a427eb33f11c415d99507"
format_stamp: "Formatted at 2026-08-09 04:54:35 on dist-test-slave-4tdj"
server_key: "36a75d93f227c44975b979c30f882c87"
server_key_iv: "a41788e54670968ddcbdcb7ac3ee120f"
server_key_version: "encryptionkey@0"
I20260809 04:54:35.815380 27646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 04:54:35.818596 27665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:35.819797 27646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:54:35.819960 27646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "46a331b3af7a427eb33f11c415d99507"
format_stamp: "Formatted at 2026-08-09 04:54:35 on dist-test-slave-4tdj"
server_key: "36a75d93f227c44975b979c30f882c87"
server_key_iv: "a41788e54670968ddcbdcb7ac3ee120f"
server_key_version: "encryptionkey@0"
I20260809 04:54:35.820142 27646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:35.830762 27646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:35.831419 27646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:35.831678 27646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:35.849049 27646 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:44321
I20260809 04:54:35.849086 27742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:44321 every 8 connection(s)
I20260809 04:54:35.850253 27646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:54:35.853780 27743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:35.858248 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 27646
I20260809 04:54:35.858433 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:54:35.858847 32116 external_mini_cluster.cc:1428] Setting key 1c8d77b9d80dee635f9353e925a206ad
I20260809 04:54:35.863652 27743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507: Bootstrap starting.
I20260809 04:54:35.866797 27743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:35.867892 27743 log.cc:824] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:35.871287 27743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507: No bootstrap required, opened a new log
I20260809 04:54:35.876166 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:35.864202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.31.93.62:44321)
0809 04:54:35.865165 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:35.865181 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:35.865239 (+ 58us) server_negotiation.cc:407] Connection header received
0809 04:54:35.866885 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:35.866911 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:35.867351 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:35.867654 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:35.868346 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:35.869938 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:35.871922 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:35.872576 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:35.872699 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:35.873942 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:35.873951 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:35.873967 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:35.874548 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:35.874637 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:35.874660 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:35.874742 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:35.874965 (+ 223us) server_negotiation.cc:299] Negotiation successful
0809 04:54:35.875243 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":333,"threads_started":1}
I20260809 04:54:35.880163 27743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a331b3af7a427eb33f11c415d99507" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44321 } }
I20260809 04:54:35.880510 27743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:35.880621 27743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46a331b3af7a427eb33f11c415d99507, State: Initialized, Role: FOLLOWER
I20260809 04:54:35.881171 27743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a331b3af7a427eb33f11c415d99507" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44321 } }
I20260809 04:54:35.881412 27743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:54:35.881559 27743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:54:35.881734 27743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:35.884253 27743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a331b3af7a427eb33f11c415d99507" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44321 } }
I20260809 04:54:35.884747 27743 leader_election.cc:302] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46a331b3af7a427eb33f11c415d99507; no voters:
I20260809 04:54:35.885293 27743 leader_election.cc:288] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:54:35.885370 27752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:54:35.886058 27752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [term 1 LEADER]: Becoming Leader. State: Replica: 46a331b3af7a427eb33f11c415d99507, State: Running, Role: LEADER
I20260809 04:54:35.886554 27752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a331b3af7a427eb33f11c415d99507" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44321 } }
I20260809 04:54:35.887336 27743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:54:35.889009 27756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46a331b3af7a427eb33f11c415d99507. Latest consensus state: current_term: 1 leader_uuid: "46a331b3af7a427eb33f11c415d99507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a331b3af7a427eb33f11c415d99507" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44321 } } }
I20260809 04:54:35.888984 27754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46a331b3af7a427eb33f11c415d99507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46a331b3af7a427eb33f11c415d99507" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44321 } } }
I20260809 04:54:35.889534 27754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:35.894382 27756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507 [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:35.896075 27761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:54:35.900934 27761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:54:35.909615 27761 catalog_manager.cc:1347] Generated new cluster ID: 5efcadc471424144b5d62cbc716e6e18
I20260809 04:54:35.909715 27761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:54:35.925180 27761 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:54:35.926818 27761 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:54:35.934340 27761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507: Generated new TSK 0
I20260809 04:54:35.935228 27761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:54:35.942951 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44321
--builtin_ntp_servers=127.31.93.20:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 04:54:36.268079 27788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:36.268347 27788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:36.268482 27788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:36.275341 27788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:36.275439 27788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:36.275578 27788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:54:36.284284 27788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:36.285422 27788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:36.287159 27788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:36.299556 27800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:36.299526 27797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:36.300544 27796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:36.300881 27788 server_base.cc:1034] running on GCE node
I20260809 04:54:36.301520 27788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:36.302452 27788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:36.303680 27788 hybrid_clock.cc:648] HybridClock initialized: now 1786251276303620 us; error 45 us; skew 500 ppm
I20260809 04:54:36.303992 27788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:54:36.306188 27788 webserver.cc:460] Webserver started at http://127.31.93.1:44819/ using document root <none> and password file <none>
I20260809 04:54:36.306860 27788 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:36.306965 27788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:36.307271 27788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:36.310141 27788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "edbb0b80bd8d4bb181ea26980d8cce59"
format_stamp: "Formatted at 2026-08-09 04:54:36 on dist-test-slave-4tdj"
server_key: "1871505a56db927cbd064f2f6d149cb9"
server_key_iv: "2088f6e17055a74b1b1d254b1dec7401"
server_key_version: "encryptionkey@0"
I20260809 04:54:36.310940 27788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "edbb0b80bd8d4bb181ea26980d8cce59"
format_stamp: "Formatted at 2026-08-09 04:54:36 on dist-test-slave-4tdj"
server_key: "1871505a56db927cbd064f2f6d149cb9"
server_key_iv: "2088f6e17055a74b1b1d254b1dec7401"
server_key_version: "encryptionkey@0"
I20260809 04:54:36.315798 27788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 04:54:36.319134 27809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:36.320407 27788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 04:54:36.320568 27788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "edbb0b80bd8d4bb181ea26980d8cce59"
format_stamp: "Formatted at 2026-08-09 04:54:36 on dist-test-slave-4tdj"
server_key: "1871505a56db927cbd064f2f6d149cb9"
server_key_iv: "2088f6e17055a74b1b1d254b1dec7401"
server_key_version: "encryptionkey@0"
I20260809 04:54:36.320766 27788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:36.345433 27788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:36.346261 27788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:36.346570 27788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:36.347558 27788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:36.349056 27788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:36.349153 27788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:36.349261 27788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:36.349339 27788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:36.379369 27788 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:42503
I20260809 04:54:36.379438 27998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:42503 every 8 connection(s)
I20260809 04:54:36.380594 27788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:54:36.381387 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 27788
I20260809 04:54:36.381559 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:54:36.382087 32116 external_mini_cluster.cc:1428] Setting key 325b7a707cf1b856972c6505473eb693
I20260809 04:54:36.384984 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44321
--builtin_ntp_servers=127.31.93.20:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 04:54:36.393344 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:36.383081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:51585 (local address 127.31.93.62:44321)
0809 04:54:36.383523 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:36.383528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:36.384922 (+ 1394us) server_negotiation.cc:407] Connection header received
0809 04:54:36.386737 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:36.386742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:36.386816 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:36.386921 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:36.387953 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:36.388955 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:36.390447 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:36.391001 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:36.391121 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:36.392523 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:36.392529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:36.392533 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:36.392553 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:36.392604 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:36.392619 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:36.392702 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:36.392939 (+ 237us) server_negotiation.cc:299] Negotiation successful
0809 04:54:36.393032 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":227,"threads_started":1}
I20260809 04:54:36.394181 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:36.383692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44321 (local address 127.31.93.1:51585)
0809 04:54:36.384750 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:36.384802 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:36.386492 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:36.387117 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:36.387127 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:36.387219 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:36.387801 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:36.387819 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:36.389098 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:36.389102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:36.390314 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:36.390331 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:36.391236 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:36.391244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:36.391532 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:36.392225 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:54:36.392227 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:36.392264 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:54:36.392340 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:36.392739 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:36.392749 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:36.392756 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 04:54:36.392911 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 04:54:36.393189 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":345,"threads_started":1}
I20260809 04:54:36.395658 27999 heartbeater.cc:347] Connected to a master server at 127.31.93.62:44321
I20260809 04:54:36.395959 27999 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:36.396715 27999 heartbeater.cc:511] Master 127.31.93.62:44321 requested a full tablet report, sending...
I20260809 04:54:36.398649 27692 ts_manager.cc:194] Registered new tserver with Master: edbb0b80bd8d4bb181ea26980d8cce59 (127.31.93.1:42503)
I20260809 04:54:36.400091 27692 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.93.1:51585
W20260809 04:54:36.702136 28004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:36.702422 28004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:36.702548 28004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:36.709481 28004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:36.709574 28004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:36.709703 28004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:54:36.718313 28004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:36.719409 28004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:36.721105 28004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:36.733150 28013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:36.733188 28018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:36.733235 28016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:36.734477 28004 server_base.cc:1034] running on GCE node
I20260809 04:54:36.735015 28004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:36.735855 28004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:36.737044 28004 hybrid_clock.cc:648] HybridClock initialized: now 1786251276737012 us; error 37 us; skew 500 ppm
I20260809 04:54:36.737325 28004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:54:36.739327 28004 webserver.cc:460] Webserver started at http://127.31.93.2:37827/ using document root <none> and password file <none>
I20260809 04:54:36.739936 28004 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:36.740031 28004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:36.740311 28004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:36.742947 28004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "0a17ec0443a04153b5e42e35f063a295"
format_stamp: "Formatted at 2026-08-09 04:54:36 on dist-test-slave-4tdj"
server_key: "77b8f308714d0fa2404fb42a345a5d69"
server_key_iv: "57213c0d61b91057fefaf3c19d41356e"
server_key_version: "encryptionkey@0"
I20260809 04:54:36.743691 28004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "0a17ec0443a04153b5e42e35f063a295"
format_stamp: "Formatted at 2026-08-09 04:54:36 on dist-test-slave-4tdj"
server_key: "77b8f308714d0fa2404fb42a345a5d69"
server_key_iv: "57213c0d61b91057fefaf3c19d41356e"
server_key_version: "encryptionkey@0"
I20260809 04:54:36.748400 28004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:54:36.751649 28024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:36.752820 28004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:54:36.752955 28004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "0a17ec0443a04153b5e42e35f063a295"
format_stamp: "Formatted at 2026-08-09 04:54:36 on dist-test-slave-4tdj"
server_key: "77b8f308714d0fa2404fb42a345a5d69"
server_key_iv: "57213c0d61b91057fefaf3c19d41356e"
server_key_version: "encryptionkey@0"
I20260809 04:54:36.753139 28004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:36.774865 28004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:36.775576 28004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:36.775827 28004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:36.776705 28004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:36.778025 28004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:36.778123 28004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:36.778226 28004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:36.778311 28004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:36.807175 28004 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:40709
I20260809 04:54:36.807231 28191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:40709 every 8 connection(s)
I20260809 04:54:36.808362 28004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:54:36.812204 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 28004
I20260809 04:54:36.812400 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:54:36.812973 32116 external_mini_cluster.cc:1428] Setting key 5d92d9225b6725886a659e001e707743
I20260809 04:54:36.815873 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44321
--builtin_ntp_servers=127.31.93.20:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 04:54:36.821096 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:36.810701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:51223 (local address 127.31.93.62:44321)
0809 04:54:36.810878 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:36.810882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:36.812259 (+ 1377us) server_negotiation.cc:407] Connection header received
0809 04:54:36.814338 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:36.814344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:36.814415 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:36.814512 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:36.815875 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:36.816632 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:36.818669 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:36.819069 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:36.819160 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:36.820449 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:36.820453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:36.820456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:36.820472 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:36.820503 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:36.820514 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:36.820575 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:36.820774 (+ 199us) server_negotiation.cc:299] Negotiation successful
0809 04:54:36.820844 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:54:36.822239 28194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:36.811173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44321 (local address 127.31.93.2:51223)
0809 04:54:36.812105 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:36.812151 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:36.814076 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:36.814854 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:36.814864 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:36.814977 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:36.815713 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:36.815737 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:36.816797 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:36.816802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:36.818528 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:36.818543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:36.819270 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:36.819276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:36.819527 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:36.820154 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:54:36.820157 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:36.820202 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:54:36.820289 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:36.820594 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:36.820603 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:36.820609 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 04:54:36.820759 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 04:54:36.821096 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":277,"threads_started":1}
I20260809 04:54:36.823560 28192 heartbeater.cc:347] Connected to a master server at 127.31.93.62:44321
I20260809 04:54:36.823846 28192 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:36.824659 28192 heartbeater.cc:511] Master 127.31.93.62:44321 requested a full tablet report, sending...
I20260809 04:54:36.825976 27692 ts_manager.cc:194] Registered new tserver with Master: 0a17ec0443a04153b5e42e35f063a295 (127.31.93.2:40709)
I20260809 04:54:36.826790 27692 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.93.2:51223
W20260809 04:54:37.138722 28195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:37.138970 28195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:37.139096 28195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:37.146522 28195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:37.146611 28195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:37.146755 28195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:54:37.155573 28195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:37.156879 28195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:37.158679 28195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:37.170729 28205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:37.170786 28207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:37.170814 28204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:37.171810 28195 server_base.cc:1034] running on GCE node
I20260809 04:54:37.172360 28195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:37.173187 28195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:37.174381 28195 hybrid_clock.cc:648] HybridClock initialized: now 1786251277174335 us; error 31 us; skew 500 ppm
I20260809 04:54:37.174660 28195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:54:37.176647 28195 webserver.cc:460] Webserver started at http://127.31.93.3:36415/ using document root <none> and password file <none>
I20260809 04:54:37.177258 28195 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:37.177356 28195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:37.177640 28195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:37.180310 28195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "8f4281b0d4284c1f8cfaa23b85927bd0"
format_stamp: "Formatted at 2026-08-09 04:54:37 on dist-test-slave-4tdj"
server_key: "600fc1bb00859d444a7220cc264ddfe5"
server_key_iv: "436e597d63827295d0459c059ade2ea3"
server_key_version: "encryptionkey@0"
I20260809 04:54:37.181031 28195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "8f4281b0d4284c1f8cfaa23b85927bd0"
format_stamp: "Formatted at 2026-08-09 04:54:37 on dist-test-slave-4tdj"
server_key: "600fc1bb00859d444a7220cc264ddfe5"
server_key_iv: "436e597d63827295d0459c059ade2ea3"
server_key_version: "encryptionkey@0"
I20260809 04:54:37.185691 28195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260809 04:54:37.188910 28217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:37.190088 28195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:54:37.190233 28195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "8f4281b0d4284c1f8cfaa23b85927bd0"
format_stamp: "Formatted at 2026-08-09 04:54:37 on dist-test-slave-4tdj"
server_key: "600fc1bb00859d444a7220cc264ddfe5"
server_key_iv: "436e597d63827295d0459c059ade2ea3"
server_key_version: "encryptionkey@0"
I20260809 04:54:37.190460 28195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:37.225498 28195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:37.226238 28195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:37.226505 28195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:37.227416 28195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:37.228768 28195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:37.228860 28195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:37.228963 28195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:37.229031 28195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:37.258252 28195 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:40941
I20260809 04:54:37.258333 28393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:40941 every 8 connection(s)
I20260809 04:54:37.259469 28195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:54:37.263026 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 28195
I20260809 04:54:37.263223 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:54:37.263785 32116 external_mini_cluster.cc:1428] Setting key 4a25eb912aafb76e60580ae60c67f5cf
I20260809 04:54:37.271492 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.261813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:38825 (local address 127.31.93.62:44321)
0809 04:54:37.261996 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:37.262000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:37.263610 (+ 1610us) server_negotiation.cc:407] Connection header received
0809 04:54:37.265579 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:37.265584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:37.265666 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:37.265756 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:37.266851 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.267581 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.269042 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.269444 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.269549 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.270741 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:37.270745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:37.270748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:37.270765 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:37.270797 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:37.270808 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:37.270878 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:37.271096 (+ 218us) server_negotiation.cc:299] Negotiation successful
0809 04:54:37.271190 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 04:54:37.272336 28398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.262367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44321 (local address 127.31.93.3:38825)
0809 04:54:37.263434 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:37.263484 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:37.265322 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:37.265973 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:37.265981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:37.266065 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:37.266688 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:37.266707 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.267720 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.267724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:37.268912 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:37.268923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.269655 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.269663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:37.269859 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.270459 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:54:37.270462 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:37.270497 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:54:37.270581 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:37.270889 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:37.270900 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:37.270907 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 04:54:37.271042 (+ 135us) client_negotiation.cc:240] Negotiation successful
0809 04:54:37.271396 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":319,"threads_started":1}
I20260809 04:54:37.273445 28394 heartbeater.cc:347] Connected to a master server at 127.31.93.62:44321
I20260809 04:54:37.273713 28394 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:37.274415 28394 heartbeater.cc:511] Master 127.31.93.62:44321 requested a full tablet report, sending...
I20260809 04:54:37.275650 27692 ts_manager.cc:194] Registered new tserver with Master: 8f4281b0d4284c1f8cfaa23b85927bd0 (127.31.93.3:40941)
I20260809 04:54:37.276382 27692 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.93.3:38825
I20260809 04:54:37.278478 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:54:37.287811 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.282386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50600 (local address 127.31.93.62:44321)
0809 04:54:37.282560 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:37.282566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:37.282953 (+ 387us) server_negotiation.cc:407] Connection header received
0809 04:54:37.283103 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:37.283108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:37.283174 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:37.283250 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:37.283736 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.284393 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.285712 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.286098 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.286203 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.287164 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:37.287168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:37.287171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:37.287188 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:37.287217 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:37.287229 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:37.287318 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:37.287443 (+ 125us) server_negotiation.cc:299] Negotiation successful
0809 04:54:37.287547 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 04:54:37.291018 27692 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:54:37.297804 27692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:54:37.318923 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.308712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42503 (local address 127.0.0.1:34542)
0809 04:54:37.311135 (+ 2423us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:37.311152 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:37.311275 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:37.311987 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:37.311995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:37.312030 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:37.312461 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:37.312472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.314548 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.314563 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:37.316731 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:37.316744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.318039 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.318045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:37.318225 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.318273 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 04:54:37.318436 (+ 163us) client_negotiation.cc:240] Negotiation successful
0809 04:54:37.318620 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2289,"thread_start_us":2161,"threads_started":1}
I20260809 04:54:37.319679 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.309229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.31.93.1:42503)
0809 04:54:37.310728 (+ 1499us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:37.310736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:37.311259 (+ 523us) server_negotiation.cc:407] Connection header received
0809 04:54:37.311417 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:37.311425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:37.311732 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:37.312707 (+ 975us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:37.312952 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.314428 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.316899 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.317848 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.317979 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.318217 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:37.319153 (+ 936us) server_negotiation.cc:299] Negotiation successful
0809 04:54:37.319329 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1340,"thread_start_us":1211,"threads_started":1}
I20260809 04:54:37.319815 28415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.308525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:40709 (local address 127.0.0.1:37244)
0809 04:54:37.309956 (+ 1431us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:37.309988 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:37.310162 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:37.313311 (+ 3149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:37.313317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:37.313343 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:37.313636 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:37.313645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.315443 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.315449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:37.317555 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:37.317570 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.319144 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.319150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:37.319335 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.319365 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 04:54:37.319421 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 04:54:37.319531 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1221,"mutex_wait_us":39,"thread_start_us":1095,"threads_started":1}
I20260809 04:54:37.320802 28417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.308877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.31.93.2:40709)
0809 04:54:37.312743 (+ 3866us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:37.312751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:37.312776 (+ 25us) server_negotiation.cc:407] Connection header received
0809 04:54:37.312863 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:37.312869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:37.313095 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:37.313794 (+ 699us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:37.313991 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.315289 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.317752 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.318936 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.320120 (+ 1184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.320326 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:37.320375 (+ 49us) server_negotiation.cc:299] Negotiation successful
0809 04:54:37.320482 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3707,"thread_start_us":3553,"threads_started":1}
I20260809 04:54:37.321630 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.310950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:40941 (local address 127.0.0.1:49728)
0809 04:54:37.312979 (+ 2029us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:37.312996 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:37.313102 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:37.314140 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:37.314144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:37.314168 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:37.314542 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:37.314555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.316381 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.316387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:37.318436 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:37.318449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.320890 (+ 2441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.320897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:37.321058 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.321081 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:54:37.321143 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 04:54:37.321252 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1923,"mutex_wait_us":61,"thread_start_us":1780,"threads_started":1}
I20260809 04:54:37.322726 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.311524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49728 (local address 127.31.93.3:40941)
0809 04:54:37.313636 (+ 2112us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:37.313644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:37.313673 (+ 29us) server_negotiation.cc:407] Connection header received
0809 04:54:37.313750 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:37.313757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:37.313955 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:37.314124 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:37.314726 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.316233 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.318643 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.320680 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.321853 (+ 1173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.322115 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:37.322186 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 04:54:37.322367 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1941,"thread_start_us":1791,"threads_started":1}
I20260809 04:54:37.323625 27890 tablet_service.cc:1467] Processing CreateTablet for tablet c1e7baa392db44d8ac32201831f0be68 (DEFAULT_TABLE table=test-table [id=40abd5ea91174b86820e474b48886e03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:37.324342 28098 tablet_service.cc:1467] Processing CreateTablet for tablet c1e7baa392db44d8ac32201831f0be68 (DEFAULT_TABLE table=test-table [id=40abd5ea91174b86820e474b48886e03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:37.325637 28098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e7baa392db44d8ac32201831f0be68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:37.325742 27890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e7baa392db44d8ac32201831f0be68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:37.326273 28297 tablet_service.cc:1467] Processing CreateTablet for tablet c1e7baa392db44d8ac32201831f0be68 (DEFAULT_TABLE table=test-table [id=40abd5ea91174b86820e474b48886e03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:37.327642 28297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e7baa392db44d8ac32201831f0be68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:37.336269 28421 tablet_bootstrap.cc:492] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295: Bootstrap starting.
I20260809 04:54:37.339593 28422 tablet_bootstrap.cc:492] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59: Bootstrap starting.
I20260809 04:54:37.340335 28421 tablet_bootstrap.cc:654] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:37.341285 28425 tablet_bootstrap.cc:492] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0: Bootstrap starting.
I20260809 04:54:37.341583 28421 log.cc:824] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:37.342929 28422 tablet_bootstrap.cc:654] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:37.344106 28422 log.cc:824] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:37.344184 28425 tablet_bootstrap.cc:654] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:37.344780 28421 tablet_bootstrap.cc:492] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295: No bootstrap required, opened a new log
I20260809 04:54:37.345032 28421 ts_tablet_manager.cc:1397] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260809 04:54:37.345266 28425 log.cc:824] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:37.347492 28422 tablet_bootstrap.cc:492] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59: No bootstrap required, opened a new log
I20260809 04:54:37.347757 28422 ts_tablet_manager.cc:1397] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 04:54:37.348711 28425 tablet_bootstrap.cc:492] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0: No bootstrap required, opened a new log
I20260809 04:54:37.348935 28425 ts_tablet_manager.cc:1397] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260809 04:54:37.354966 28421 raft_consensus.cc:348] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a17ec0443a04153b5e42e35f063a295" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40709 } } peers { permanent_uuid: "edbb0b80bd8d4bb181ea26980d8cce59" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42503 } } peers { permanent_uuid: "8f4281b0d4284c1f8cfaa23b85927bd0" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40941 } }
I20260809 04:54:37.355407 28421 raft_consensus.cc:374] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:37.355543 28421 raft_consensus.cc:729] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a17ec0443a04153b5e42e35f063a295, State: Initialized, Role: FOLLOWER
I20260809 04:54:37.355558 28425 raft_consensus.cc:348] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a17ec0443a04153b5e42e35f063a295" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40709 } } peers { permanent_uuid: "edbb0b80bd8d4bb181ea26980d8cce59" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42503 } } peers { permanent_uuid: "8f4281b0d4284c1f8cfaa23b85927bd0" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40941 } }
I20260809 04:54:37.356029 28425 raft_consensus.cc:374] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:37.356132 28425 raft_consensus.cc:729] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4281b0d4284c1f8cfaa23b85927bd0, State: Initialized, Role: FOLLOWER
I20260809 04:54:37.356241 28421 consensus_queue.cc:260] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a17ec0443a04153b5e42e35f063a295" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40709 } } peers { permanent_uuid: "edbb0b80bd8d4bb181ea26980d8cce59" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42503 } } peers { permanent_uuid: "8f4281b0d4284c1f8cfaa23b85927bd0" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40941 } }
I20260809 04:54:37.356812 28425 consensus_queue.cc:260] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a17ec0443a04153b5e42e35f063a295" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40709 } } peers { permanent_uuid: "edbb0b80bd8d4bb181ea26980d8cce59" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42503 } } peers { permanent_uuid: "8f4281b0d4284c1f8cfaa23b85927bd0" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40941 } }
I20260809 04:54:37.357846 28422 raft_consensus.cc:348] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a17ec0443a04153b5e42e35f063a295" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40709 } } peers { permanent_uuid: "edbb0b80bd8d4bb181ea26980d8cce59" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42503 } } peers { permanent_uuid: "8f4281b0d4284c1f8cfaa23b85927bd0" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40941 } }
I20260809 04:54:37.358340 28422 raft_consensus.cc:374] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:37.358479 28422 raft_consensus.cc:729] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edbb0b80bd8d4bb181ea26980d8cce59, State: Initialized, Role: FOLLOWER
I20260809 04:54:37.359021 28394 heartbeater.cc:503] Master 127.31.93.62:44321 was elected leader, sending a full tablet report...
I20260809 04:54:37.359205 28422 consensus_queue.cc:260] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a17ec0443a04153b5e42e35f063a295" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40709 } } peers { permanent_uuid: "edbb0b80bd8d4bb181ea26980d8cce59" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42503 } } peers { permanent_uuid: "8f4281b0d4284c1f8cfaa23b85927bd0" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40941 } }
I20260809 04:54:37.359527 28425 ts_tablet_manager.cc:1428] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0: Time spent starting tablet: real 0.010s user 0.009s sys 0.002s
I20260809 04:54:37.360251 27999 heartbeater.cc:503] Master 127.31.93.62:44321 was elected leader, sending a full tablet report...
I20260809 04:54:37.362202 28422 ts_tablet_manager.cc:1428] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260809 04:54:37.364249 28192 heartbeater.cc:503] Master 127.31.93.62:44321 was elected leader, sending a full tablet report...
I20260809 04:54:37.364854 28421 ts_tablet_manager.cc:1428] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295: Time spent starting tablet: real 0.020s user 0.014s sys 0.005s
W20260809 04:54:37.382684 28000 tablet.cc:2369] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:54:37.510965 28395 tablet.cc:2369] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:54:37.557003 28430 raft_consensus.cc:482] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:54:37.557324 28430 raft_consensus.cc:504] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a17ec0443a04153b5e42e35f063a295" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40709 } } peers { permanent_uuid: "edbb0b80bd8d4bb181ea26980d8cce59" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42503 } } peers { permanent_uuid: "8f4281b0d4284c1f8cfaa23b85927bd0" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40941 } }
I20260809 04:54:37.559149 28430 leader_election.cc:288] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edbb0b80bd8d4bb181ea26980d8cce59 (127.31.93.1:42503), 8f4281b0d4284c1f8cfaa23b85927bd0 (127.31.93.3:40941)
W20260809 04:54:37.560217 28193 tablet.cc:2369] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:54:37.566097 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.559471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:46639 (local address 127.31.93.1:42503)
0809 04:54:37.559799 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:37.559807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:37.560725 (+ 918us) server_negotiation.cc:407] Connection header received
0809 04:54:37.560905 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:37.560910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:37.560998 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:37.561085 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:37.561705 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.562435 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.564388 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.565125 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.565249 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.565337 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:37.565723 (+ 386us) server_negotiation.cc:299] Negotiation successful
0809 04:54:37.565842 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260809 04:54:37.566102 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.559731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:47527 (local address 127.31.93.3:40941)
0809 04:54:37.559905 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:37.559912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:37.560796 (+ 884us) server_negotiation.cc:407] Connection header received
0809 04:54:37.561000 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:37.561006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:37.561080 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:37.561160 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:37.561741 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.562459 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.564403 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.565163 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.565287 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.565403 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:37.565736 (+ 333us) server_negotiation.cc:299] Negotiation successful
0809 04:54:37.565852 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 04:54:37.566183 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.559446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:40941 (local address 127.31.93.2:47527)
0809 04:54:37.560680 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:37.560704 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:37.560853 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:37.561187 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:37.561192 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:37.561219 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:37.561592 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:37.561601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.562576 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.562581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:37.564199 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:37.564211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.565369 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.565376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:37.565547 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.565610 (+ 63us) client_negotiation.cc:769] Sending connection context
0809 04:54:37.565684 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 04:54:37.565812 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"mutex_wait_us":86,"thread_start_us":566,"threads_started":1}
I20260809 04:54:37.566666 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.559325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42503 (local address 127.31.93.2:46639)
0809 04:54:37.560605 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:37.560626 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:37.560775 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:37.561131 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:37.561136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:37.561161 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:37.561553 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:37.561566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.562561 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.562564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:37.564192 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:37.564205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.565349 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.565356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:37.565501 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.565604 (+ 103us) client_negotiation.cc:769] Sending connection context
0809 04:54:37.565682 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 04:54:37.565812 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":137,"thread_start_us":364,"threads_started":1}
I20260809 04:54:37.567138 28326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e7baa392db44d8ac32201831f0be68" candidate_uuid: "0a17ec0443a04153b5e42e35f063a295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4281b0d4284c1f8cfaa23b85927bd0" is_pre_election: true
I20260809 04:54:37.567711 28326 raft_consensus.cc:2454] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a17ec0443a04153b5e42e35f063a295 in term 0.
I20260809 04:54:37.567616 27933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e7baa392db44d8ac32201831f0be68" candidate_uuid: "0a17ec0443a04153b5e42e35f063a295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edbb0b80bd8d4bb181ea26980d8cce59" is_pre_election: true
I20260809 04:54:37.568145 27933 raft_consensus.cc:2454] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a17ec0443a04153b5e42e35f063a295 in term 0.
I20260809 04:54:37.568390 28031 leader_election.cc:302] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a17ec0443a04153b5e42e35f063a295, 8f4281b0d4284c1f8cfaa23b85927bd0; no voters:
I20260809 04:54:37.568768 28430 raft_consensus.cc:2781] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:54:37.568892 28430 raft_consensus.cc:482] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:54:37.568988 28430 raft_consensus.cc:3037] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:37.571157 28430 raft_consensus.cc:504] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a17ec0443a04153b5e42e35f063a295" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40709 } } peers { permanent_uuid: "edbb0b80bd8d4bb181ea26980d8cce59" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42503 } } peers { permanent_uuid: "8f4281b0d4284c1f8cfaa23b85927bd0" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40941 } }
I20260809 04:54:37.571753 28430 leader_election.cc:288] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [CANDIDATE]: Term 1 election: Requested vote from peers edbb0b80bd8d4bb181ea26980d8cce59 (127.31.93.1:42503), 8f4281b0d4284c1f8cfaa23b85927bd0 (127.31.93.3:40941)
I20260809 04:54:37.572082 27933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e7baa392db44d8ac32201831f0be68" candidate_uuid: "0a17ec0443a04153b5e42e35f063a295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edbb0b80bd8d4bb181ea26980d8cce59"
I20260809 04:54:37.572190 28326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e7baa392db44d8ac32201831f0be68" candidate_uuid: "0a17ec0443a04153b5e42e35f063a295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4281b0d4284c1f8cfaa23b85927bd0"
I20260809 04:54:37.572274 27933 raft_consensus.cc:3037] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:37.572372 28326 raft_consensus.cc:3037] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:37.575356 28326 raft_consensus.cc:2454] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a17ec0443a04153b5e42e35f063a295 in term 1.
I20260809 04:54:37.575356 27933 raft_consensus.cc:2454] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a17ec0443a04153b5e42e35f063a295 in term 1.
I20260809 04:54:37.575726 28031 leader_election.cc:302] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a17ec0443a04153b5e42e35f063a295, 8f4281b0d4284c1f8cfaa23b85927bd0; no voters:
I20260809 04:54:37.576013 28430 raft_consensus.cc:2781] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:54:37.576407 28430 raft_consensus.cc:686] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [term 1 LEADER]: Becoming Leader. State: Replica: 0a17ec0443a04153b5e42e35f063a295, State: Running, Role: LEADER
I20260809 04:54:37.576843 28430 consensus_queue.cc:237] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a17ec0443a04153b5e42e35f063a295" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40709 } } peers { permanent_uuid: "edbb0b80bd8d4bb181ea26980d8cce59" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42503 } } peers { permanent_uuid: "8f4281b0d4284c1f8cfaa23b85927bd0" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40941 } }
I20260809 04:54:37.581097 27692 catalog_manager.cc:5495] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a17ec0443a04153b5e42e35f063a295 (127.31.93.2). New cstate: current_term: 1 leader_uuid: "0a17ec0443a04153b5e42e35f063a295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a17ec0443a04153b5e42e35f063a295" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 40709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edbb0b80bd8d4bb181ea26980d8cce59" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f4281b0d4284c1f8cfaa23b85927bd0" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40941 } health_report { overall_health: UNKNOWN } } }
I20260809 04:54:37.607860 28417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.603175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37246 (local address 127.31.93.2:40709)
0809 04:54:37.603327 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:37.603332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:37.603456 (+ 124us) server_negotiation.cc:407] Connection header received
0809 04:54:37.603636 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:37.603640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:37.603712 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:37.603826 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:54:37.604135 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.604801 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.606129 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.606542 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.606632 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.607036 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:54:37.607479 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:54:37.607592 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:37.607626 (+ 34us) server_negotiation.cc:299] Negotiation successful
0809 04:54:37.607685 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 04:54:37.615726 28326 raft_consensus.cc:1261] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0 [term 1 FOLLOWER]: Refusing update from remote peer 0a17ec0443a04153b5e42e35f063a295: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:54:37.615731 27933 raft_consensus.cc:1261] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59 [term 1 FOLLOWER]: Refusing update from remote peer 0a17ec0443a04153b5e42e35f063a295: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:54:37.616750 28430 consensus_queue.cc:1035] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4281b0d4284c1f8cfaa23b85927bd0" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:54:37.617043 28440 consensus_queue.cc:1035] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edbb0b80bd8d4bb181ea26980d8cce59" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 42503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:54:37.641260 27692 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:54:37.641525 27692 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:54:37.644989 27692 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46a331b3af7a427eb33f11c415d99507: Sending DeleteTablet for 3 replicas of tablet c1e7baa392db44d8ac32201831f0be68
I20260809 04:54:37.645977 28098 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e7baa392db44d8ac32201831f0be68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:37 UTC) from {username='slave'} at 127.0.0.1:37244
I20260809 04:54:37.646128 27890 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e7baa392db44d8ac32201831f0be68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:37 UTC) from {username='slave'} at 127.0.0.1:34542
I20260809 04:54:37.646940 28460 tablet_replica.cc:330] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295: stopping tablet replica
I20260809 04:54:37.647634 28297 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e7baa392db44d8ac32201831f0be68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:37 UTC) from {username='slave'} at 127.0.0.1:49728
I20260809 04:54:37.648168 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 27788
I20260809 04:54:37.653518 28460 raft_consensus.cc:2229] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:54:37.654369 28460 raft_consensus.cc:2258] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:37.658743 28460 ts_tablet_manager.cc:1905] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:54:37.663869 28463 tablet_replica.cc:330] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0: stopping tablet replica
I20260809 04:54:37.664404 28463 raft_consensus.cc:2229] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:54:37.664949 28463 raft_consensus.cc:2258] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:37.665799 28462 tablet_replica.cc:330] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59: stopping tablet replica
I20260809 04:54:37.668092 28463 ts_tablet_manager.cc:1905] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:54:37.670591 28460 ts_tablet_manager.cc:1918] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:54:37.670832 28460 log.cc:1196] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/c1e7baa392db44d8ac32201831f0be68
I20260809 04:54:37.671433 28460 ts_tablet_manager.cc:1939] T c1e7baa392db44d8ac32201831f0be68 P 0a17ec0443a04153b5e42e35f063a295: Deleting consensus metadata
I20260809 04:54:37.672684 28463 ts_tablet_manager.cc:1918] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:54:37.672821 28463 log.cc:1196] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/c1e7baa392db44d8ac32201831f0be68
I20260809 04:54:37.668504 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.649609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.31.93.1:42503)
0809 04:54:37.649790 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:37.649796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:37.659955 (+ 10159us) server_negotiation.cc:407] Connection header received
0809 04:54:37.660385 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:37.660391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:37.660496 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:37.660592 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:37.661247 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.662595 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.664742 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.665493 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.665622 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.666875 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:37.666885 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:37.666901 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:37.667632 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:37.667690 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:37.667714 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:37.667822 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:37.668028 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 04:54:37.668152 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:54:37.673207 27670 catalog_manager.cc:4841] TS 0a17ec0443a04153b5e42e35f063a295 (127.31.93.2:40709): tablet c1e7baa392db44d8ac32201831f0be68 (table test-table [id=40abd5ea91174b86820e474b48886e03]) successfully deleted
I20260809 04:54:37.673333 28462 raft_consensus.cc:2229] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:54:37.673588 27994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:37.674327 28463 ts_tablet_manager.cc:1939] T c1e7baa392db44d8ac32201831f0be68 P 8f4281b0d4284c1f8cfaa23b85927bd0: Deleting consensus metadata
I20260809 04:54:37.676023 27670 catalog_manager.cc:4841] TS 8f4281b0d4284c1f8cfaa23b85927bd0 (127.31.93.3:40941): tablet c1e7baa392db44d8ac32201831f0be68 (table test-table [id=40abd5ea91174b86820e474b48886e03]) successfully deleted
I20260809 04:54:37.761713 28462 raft_consensus.cc:2258] T c1e7baa392db44d8ac32201831f0be68 P edbb0b80bd8d4bb181ea26980d8cce59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:37.762390 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 27788
I20260809 04:54:37.785593 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 28004
W20260809 04:54:37.785530 27667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.93.1:42503 (error 108)
I20260809 04:54:37.787678 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.787120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42503 (local address 127.0.0.1:58558)
0809 04:54:37.787364 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:37.787488 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260809 04:54:37.787917 27667 catalog_manager.cc:4584] TS edbb0b80bd8d4bb181ea26980d8cce59 (127.31.93.1:42503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c1e7baa392db44d8ac32201831f0be68: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42503: connect: Connection refused (error 111)
I20260809 04:54:37.793126 28417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.786846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39470 (local address 127.31.93.2:40709)
0809 04:54:37.787014 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:37.787019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:37.787054 (+ 35us) server_negotiation.cc:407] Connection header received
0809 04:54:37.787208 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:37.787216 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:37.787323 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:37.787417 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:37.787919 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.788874 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.790140 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.790636 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.790751 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.791795 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:37.791804 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:37.791822 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:37.792393 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:37.792444 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:37.792468 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:37.792569 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:37.792713 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 04:54:37.792811 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 04:54:37.793877 28187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:37.841408 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.840969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42503 (local address 127.0.0.1:58566)
0809 04:54:37.841173 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:37.841262 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 04:54:37.842458 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.842080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42503 (local address 127.0.0.1:58582)
0809 04:54:37.842228 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:37.842330 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260809 04:54:37.901530 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 28004
I20260809 04:54:37.902916 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.902443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42503 (local address 127.0.0.1:58588)
0809 04:54:37.902637 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:37.902733 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 04:54:37.903872 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.903532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42503 (local address 127.0.0.1:58596)
0809 04:54:37.903685 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:37.903758 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260809 04:54:37.925455 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 28195
I20260809 04:54:37.932796 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.926229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45280 (local address 127.31.93.3:40941)
0809 04:54:37.926439 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:37.926447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:37.926530 (+ 83us) server_negotiation.cc:407] Connection header received
0809 04:54:37.926752 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:37.926759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:37.926856 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:37.926953 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:37.927415 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.928165 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.929594 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:37.930027 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:37.930151 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:37.931384 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:37.931394 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:37.931409 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:37.931992 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:37.932043 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:37.932065 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:37.932178 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:37.932319 (+ 141us) server_negotiation.cc:299] Negotiation successful
0809 04:54:37.932438 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 04:54:37.933429 28387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:37.984453 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.983932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42503 (local address 127.0.0.1:58610)
0809 04:54:37.984165 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:37.984290 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260809 04:54:37.985504 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:37.985114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:42503 (local address 127.0.0.1:58616)
0809 04:54:37.985314 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:37.985396 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.1:42503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260809 04:54:38.019693 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 28195
I20260809 04:54:38.037528 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 27646
I20260809 04:54:38.038580 27736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:38.144377 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 27646
2026-08-09T04:54:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2777 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:54:38 dist-test-slave-4tdj krb5kdc[28476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:54:38 dist-test-slave-4tdj krb5kdc[28476](info): set up 2 sockets
Aug 09 04:54:38 dist-test-slave-4tdj krb5kdc[28476](info): commencing operation
krb5kdc: starting...
W20260809 04:54:39.356753 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:54:39 dist-test-slave-4tdj krb5kdc[28476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:54:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:54:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:54:39.600979 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:35693
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:35693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:39.926330 28500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:39.926637 28500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:39.926782 28500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:39.934075 28500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:54:39.934186 28500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:39.934274 28500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:39.934409 28500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:54:39.934486 28500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:54:39.944600 28500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:35693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:35693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:39.945962 28500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:39.947381 28500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:39.959126 28509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:39.959257 28507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:39.959623 28506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:39.960108 28500 server_base.cc:1034] running on GCE node
I20260809 04:54:39.960929 28500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:39.962402 28500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:39.963610 28500 hybrid_clock.cc:648] HybridClock initialized: now 1786251279963553 us; error 46 us; skew 500 ppm
I20260809 04:54:39.963973 28500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:39 dist-test-slave-4tdj krb5kdc[28476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251279, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:39.967140 28500 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:54:39.968585 28500 webserver.cc:460] Webserver started at http://127.31.93.62:33157/ using document root <none> and password file <none>
I20260809 04:54:39.969216 28500 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:39.969316 28500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:39.969558 28500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:39.972532 28500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "eefa6c2fae7847bfb487c80324e20397"
format_stamp: "Formatted at 2026-08-09 04:54:39 on dist-test-slave-4tdj"
server_key: "7139e92a5a14df12fb0498ef54e415fe"
server_key_iv: "2ebc80ac66ec1ca6841c4f370535b218"
server_key_version: "encryptionkey@0"
I20260809 04:54:39.973219 28500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "eefa6c2fae7847bfb487c80324e20397"
format_stamp: "Formatted at 2026-08-09 04:54:39 on dist-test-slave-4tdj"
server_key: "7139e92a5a14df12fb0498ef54e415fe"
server_key_iv: "2ebc80ac66ec1ca6841c4f370535b218"
server_key_version: "encryptionkey@0"
I20260809 04:54:39.978094 28500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 04:54:39.981427 28519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:39.982758 28500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:54:39.982920 28500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "eefa6c2fae7847bfb487c80324e20397"
format_stamp: "Formatted at 2026-08-09 04:54:39 on dist-test-slave-4tdj"
server_key: "7139e92a5a14df12fb0498ef54e415fe"
server_key_iv: "2ebc80ac66ec1ca6841c4f370535b218"
server_key_version: "encryptionkey@0"
I20260809 04:54:39.983096 28500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:40.001705 28500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:40.005689 28500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:40.005939 28500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:40.029031 28500 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:35693
I20260809 04:54:40.029060 28602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:35693 every 8 connection(s)
I20260809 04:54:40.030432 28500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:54:40.032788 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 28500
I20260809 04:54:40.032967 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:54:40.033399 32116 external_mini_cluster.cc:1428] Setting key 5b13c300703ef538d12eb2c57ece3fd4
I20260809 04:54:40.034356 28603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 04:54:40 dist-test-slave-4tdj krb5kdc[28476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:54:40.048385 28603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397: Bootstrap starting.
I20260809 04:54:40.051760 28603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:40.052966 28603 log.cc:824] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:40.056092 28603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397: No bootstrap required, opened a new log
I20260809 04:54:40.056144 28606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:40.035151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47578 (local address 127.31.93.62:35693)
0809 04:54:40.035910 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:40.035921 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:40.035964 (+ 43us) server_negotiation.cc:407] Connection header received
0809 04:54:40.037574 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:40.037595 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:40.037983 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:40.038361 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:40.040104 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:40.041434 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:40.043244 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:40.043876 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:40.043997 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:40.049032 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:40.049039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:40.049054 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:40.049089 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:40.052115 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:40.052885 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:40.052891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:40.052896 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:40.053053 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:40.053432 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:40.053435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:40.053438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:40.054181 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:40.054467 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:40.054856 (+ 389us) server_negotiation.cc:299] Negotiation successful
0809 04:54:40.055187 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":330,"threads_started":1}
I20260809 04:54:40.063812 28603 raft_consensus.cc:348] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefa6c2fae7847bfb487c80324e20397" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 35693 } }
I20260809 04:54:40.064149 28603 raft_consensus.cc:374] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:40.064242 28603 raft_consensus.cc:729] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eefa6c2fae7847bfb487c80324e20397, State: Initialized, Role: FOLLOWER
I20260809 04:54:40.064728 28603 consensus_queue.cc:260] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefa6c2fae7847bfb487c80324e20397" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 35693 } }
I20260809 04:54:40.064934 28603 raft_consensus.cc:388] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:54:40.065055 28603 raft_consensus.cc:482] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:54:40.065240 28603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:40.067449 28603 raft_consensus.cc:504] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefa6c2fae7847bfb487c80324e20397" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 35693 } }
I20260809 04:54:40.067898 28603 leader_election.cc:302] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eefa6c2fae7847bfb487c80324e20397; no voters:
I20260809 04:54:40.068449 28603 leader_election.cc:288] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:54:40.068552 28609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:54:40.069157 28609 raft_consensus.cc:686] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [term 1 LEADER]: Becoming Leader. State: Replica: eefa6c2fae7847bfb487c80324e20397, State: Running, Role: LEADER
I20260809 04:54:40.069976 28609 consensus_queue.cc:237] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefa6c2fae7847bfb487c80324e20397" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 35693 } }
I20260809 04:54:40.070444 28603 sys_catalog.cc:564] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:54:40.072783 28611 sys_catalog.cc:455] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eefa6c2fae7847bfb487c80324e20397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefa6c2fae7847bfb487c80324e20397" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 35693 } } }
I20260809 04:54:40.073081 28611 sys_catalog.cc:458] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:40.074570 28621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:54:40.074892 28612 sys_catalog.cc:455] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eefa6c2fae7847bfb487c80324e20397. Latest consensus state: current_term: 1 leader_uuid: "eefa6c2fae7847bfb487c80324e20397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefa6c2fae7847bfb487c80324e20397" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 35693 } } }
I20260809 04:54:40.075153 28612 sys_catalog.cc:458] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397 [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:40.079388 28621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:54:40.087862 28621 catalog_manager.cc:1347] Generated new cluster ID: 3139380f4b1342f2aca5f89bb2d3fcfc
I20260809 04:54:40.087955 28621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:54:40.101987 28621 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:54:40.103495 28621 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:54:40.110903 28621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397: Generated new TSK 0
I20260809 04:54:40.111635 28621 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:54:40.175422 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:35693
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:40.497087 28644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:40.497326 28644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:40.497454 28644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:40.504278 28644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:40.504383 28644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:40.504520 28644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:54:40.513646 28644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:35693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:40.514865 28644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:40.516103 28644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:40.527864 28654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:40.527905 28649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:40.527943 28656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:40.528889 28644 server_base.cc:1034] running on GCE node
I20260809 04:54:40.529387 28644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:40.530182 28644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:40.531373 28644 hybrid_clock.cc:648] HybridClock initialized: now 1786251280531337 us; error 47 us; skew 500 ppm
I20260809 04:54:40.531636 28644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:40 dist-test-slave-4tdj krb5kdc[28476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251280, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:40.534458 28644 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:54:40.535739 28644 webserver.cc:460] Webserver started at http://127.31.93.1:44663/ using document root <none> and password file <none>
I20260809 04:54:40.536324 28644 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:40.536438 28644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:40.536725 28644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:40.539369 28644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "b02a0a02ed844e1da9b3bc6a17d4bab5"
format_stamp: "Formatted at 2026-08-09 04:54:40 on dist-test-slave-4tdj"
server_key: "14a75a86929150bdf7a1bf1aafaa062f"
server_key_iv: "912a50473b2b999e95ec9f80f7bcf018"
server_key_version: "encryptionkey@0"
I20260809 04:54:40.540060 28644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "b02a0a02ed844e1da9b3bc6a17d4bab5"
format_stamp: "Formatted at 2026-08-09 04:54:40 on dist-test-slave-4tdj"
server_key: "14a75a86929150bdf7a1bf1aafaa062f"
server_key_iv: "912a50473b2b999e95ec9f80f7bcf018"
server_key_version: "encryptionkey@0"
I20260809 04:54:40.544683 28644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:54:40.547849 28666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:40.548985 28644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:54:40.549131 28644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "b02a0a02ed844e1da9b3bc6a17d4bab5"
format_stamp: "Formatted at 2026-08-09 04:54:40 on dist-test-slave-4tdj"
server_key: "14a75a86929150bdf7a1bf1aafaa062f"
server_key_iv: "912a50473b2b999e95ec9f80f7bcf018"
server_key_version: "encryptionkey@0"
I20260809 04:54:40.549288 28644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:40.566484 28644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:40.569722 28644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:40.569952 28644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:40.570839 28644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:40.572086 28644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:40.572161 28644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:40.572270 28644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:40.572321 28644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:40.600871 28644 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:41613
I20260809 04:54:40.600924 28841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:41613 every 8 connection(s)
I20260809 04:54:40.601989 28644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:54:40.610133 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 28644
I20260809 04:54:40.610327 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:54:40.610855 32116 external_mini_cluster.cc:1428] Setting key 3e8d70acb8bb7a97dd8b953085802c05
Aug 09 04:54:40 dist-test-slave-4tdj krb5kdc[28476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251280, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:54:40.620388 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:40.604235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:46145 (local address 127.31.93.62:35693)
0809 04:54:40.604660 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:40.604666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:40.605613 (+ 947us) server_negotiation.cc:407] Connection header received
0809 04:54:40.607314 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:40.607318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:40.607380 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:40.607469 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:40.608750 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:40.609403 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:40.611147 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:40.611650 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:40.611749 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:40.615288 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:40.615293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:40.615297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:40.615327 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:40.617636 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:40.618399 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:40.618402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:40.618405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:40.618520 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:40.618931 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:40.618934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:40.618936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:40.619187 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:40.619292 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:40.619967 (+ 675us) server_negotiation.cc:299] Negotiation successful
0809 04:54:40.620130 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":193,"threads_started":1}
I20260809 04:54:40.621511 28848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:40.604764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:35693 (local address 127.31.93.1:46145)
0809 04:54:40.605453 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:40.605500 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:40.607095 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:40.607640 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:40.607647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:40.608126 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:40.608626 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:40.608643 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:40.609517 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:40.609520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:40.610823 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:40.610835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:40.611878 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:40.611886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:40.612068 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:40.612618 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:54:40.612644 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:40.615058 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:40.617767 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:40.617774 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:40.617787 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:40.618288 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:40.618621 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:40.618625 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:40.618628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:40.618845 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:40.619298 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:40.619305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:40.619653 (+ 348us) client_negotiation.cc:769] Sending connection context
0809 04:54:40.619915 (+ 262us) client_negotiation.cc:240] Negotiation successful
0809 04:54:40.620363 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":274,"threads_started":1}
I20260809 04:54:40.622781 28842 heartbeater.cc:347] Connected to a master server at 127.31.93.62:35693
I20260809 04:54:40.623029 28842 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:40.623675 28842 heartbeater.cc:511] Master 127.31.93.62:35693 requested a full tablet report, sending...
I20260809 04:54:40.625813 28550 ts_manager.cc:194] Registered new tserver with Master: b02a0a02ed844e1da9b3bc6a17d4bab5 (127.31.93.1:41613)
I20260809 04:54:40.627709 28550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:46145
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:54:40.666662 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:35693
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:40.991480 28852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:40.991736 28852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:40.991868 28852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:40.998822 28852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:40.998924 28852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:40.999085 28852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:54:41.008323 28852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:35693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:41.009492 28852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:41.010783 28852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:41.026732 28859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:41.026681 28862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:41.027076 28864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:41.028288 28852 server_base.cc:1034] running on GCE node
I20260809 04:54:41.028898 28852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:41.029817 28852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:41.031009 28852 hybrid_clock.cc:648] HybridClock initialized: now 1786251281030942 us; error 62 us; skew 500 ppm
I20260809 04:54:41.031306 28852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:41 dist-test-slave-4tdj krb5kdc[28476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251281, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:41.037102 28852 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:54:41.038563 28852 webserver.cc:460] Webserver started at http://127.31.93.2:44577/ using document root <none> and password file <none>
I20260809 04:54:41.039240 28852 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:41.039342 28852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:41.039713 28852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:41.042512 28852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "e70d7478178a4bf59123fb8480d32411"
format_stamp: "Formatted at 2026-08-09 04:54:41 on dist-test-slave-4tdj"
server_key: "d63e5e45dcba381dcd6cbe392c2100ad"
server_key_iv: "68baf57a89fb60bcdb8fe1c7dbb3c2e0"
server_key_version: "encryptionkey@0"
I20260809 04:54:41.043226 28852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "e70d7478178a4bf59123fb8480d32411"
format_stamp: "Formatted at 2026-08-09 04:54:41 on dist-test-slave-4tdj"
server_key: "d63e5e45dcba381dcd6cbe392c2100ad"
server_key_iv: "68baf57a89fb60bcdb8fe1c7dbb3c2e0"
server_key_version: "encryptionkey@0"
I20260809 04:54:41.048055 28852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:54:41.051301 28874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:41.052587 28852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:54:41.052722 28852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "e70d7478178a4bf59123fb8480d32411"
format_stamp: "Formatted at 2026-08-09 04:54:41 on dist-test-slave-4tdj"
server_key: "d63e5e45dcba381dcd6cbe392c2100ad"
server_key_iv: "68baf57a89fb60bcdb8fe1c7dbb3c2e0"
server_key_version: "encryptionkey@0"
I20260809 04:54:41.052891 28852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:41.064414 28852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:41.068204 28852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:41.068464 28852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:41.069361 28852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:41.070706 28852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:41.070779 28852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:41.070888 28852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:41.070937 28852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:41.100679 28852 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:43189
I20260809 04:54:41.100726 29032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:43189 every 8 connection(s)
I20260809 04:54:41.101853 28852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:54:41.107164 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 28852
I20260809 04:54:41.107364 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:54:41.107964 32116 external_mini_cluster.cc:1428] Setting key fc14746ff6901237e7469413060b2a87
Aug 09 04:54:41 dist-test-slave-4tdj krb5kdc[28476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251281, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:54:41.121371 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.104290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:46501 (local address 127.31.93.62:35693)
0809 04:54:41.104461 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:41.104465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:41.105571 (+ 1106us) server_negotiation.cc:407] Connection header received
0809 04:54:41.107474 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:41.107481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:41.107559 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:41.107646 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:41.109440 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.110271 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.112229 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.112635 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.112743 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.116100 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:41.116105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:41.116109 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:41.116140 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:41.118580 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:41.119228 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:41.119232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:41.119234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:41.119371 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:41.119770 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:41.119774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:41.119776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:41.120026 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:41.120164 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:41.120816 (+ 652us) server_negotiation.cc:299] Negotiation successful
0809 04:54:41.121036 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 04:54:41.122256 29035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.104719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:35693 (local address 127.31.93.2:46501)
0809 04:54:41.105408 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:41.105444 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:41.107178 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:41.107909 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:41.107920 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:41.108641 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:41.109277 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:41.109294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.110460 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.110466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:41.112089 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:41.112102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.112844 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.112852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:41.113083 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.113676 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:54:41.113698 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:41.115888 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:41.118705 (+ 2817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:41.118713 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:41.118727 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:41.119119 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:41.119472 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:41.119477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:41.119482 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:41.119665 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:41.120154 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:41.120161 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:41.120465 (+ 304us) client_negotiation.cc:769] Sending connection context
0809 04:54:41.120701 (+ 236us) client_negotiation.cc:240] Negotiation successful
0809 04:54:41.121170 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":265,"threads_started":1}
I20260809 04:54:41.123759 29033 heartbeater.cc:347] Connected to a master server at 127.31.93.62:35693
I20260809 04:54:41.124028 29033 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:41.124693 29033 heartbeater.cc:511] Master 127.31.93.62:35693 requested a full tablet report, sending...
I20260809 04:54:41.126080 28550 ts_manager.cc:194] Registered new tserver with Master: e70d7478178a4bf59123fb8480d32411 (127.31.93.2:43189)
I20260809 04:54:41.127156 28550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:46501
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:54:41.166868 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:35693
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:41.483328 29038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:41.483587 29038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:41.483680 29038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:41.490844 29038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:41.490939 29038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:41.491078 29038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:54:41.500329 29038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:35693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:41.501439 29038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:41.502686 29038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:41.514906 29048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:41.514824 29050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:41.515197 29052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:41.515744 29038 server_base.cc:1034] running on GCE node
I20260809 04:54:41.516278 29038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:41.517100 29038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:41.518311 29038 hybrid_clock.cc:648] HybridClock initialized: now 1786251281518230 us; error 50 us; skew 500 ppm
I20260809 04:54:41.518623 29038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:41 dist-test-slave-4tdj krb5kdc[28476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251281, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:41.521538 29038 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:54:41.522854 29038 webserver.cc:460] Webserver started at http://127.31.93.3:37213/ using document root <none> and password file <none>
I20260809 04:54:41.523484 29038 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:41.523594 29038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:41.523880 29038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:41.526635 29038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "027064c7b7ec47129b426bbe18272a7c"
format_stamp: "Formatted at 2026-08-09 04:54:41 on dist-test-slave-4tdj"
server_key: "fcbd9734da1d0612bcacc70bcf353e01"
server_key_iv: "784a976609661e88b51b5157f2f60844"
server_key_version: "encryptionkey@0"
I20260809 04:54:41.527349 29038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "027064c7b7ec47129b426bbe18272a7c"
format_stamp: "Formatted at 2026-08-09 04:54:41 on dist-test-slave-4tdj"
server_key: "fcbd9734da1d0612bcacc70bcf353e01"
server_key_iv: "784a976609661e88b51b5157f2f60844"
server_key_version: "encryptionkey@0"
I20260809 04:54:41.532063 29038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:54:41.535233 29062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:41.536399 29038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:54:41.536551 29038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "027064c7b7ec47129b426bbe18272a7c"
format_stamp: "Formatted at 2026-08-09 04:54:41 on dist-test-slave-4tdj"
server_key: "fcbd9734da1d0612bcacc70bcf353e01"
server_key_iv: "784a976609661e88b51b5157f2f60844"
server_key_version: "encryptionkey@0"
I20260809 04:54:41.536720 29038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:41.552309 29038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:41.556180 29038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:41.556459 29038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:41.557466 29038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:41.558754 29038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:41.558825 29038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:41.558912 29038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:41.558955 29038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:41.588374 29038 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:34797
I20260809 04:54:41.588428 29235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:34797 every 8 connection(s)
I20260809 04:54:41.589579 29038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:54:41.590112 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 29038
I20260809 04:54:41.590318 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:54:41.590884 32116 external_mini_cluster.cc:1428] Setting key d697bd1ef0372c389686ed21e51f142b
Aug 09 04:54:41 dist-test-slave-4tdj krb5kdc[28476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251281, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:54:41.609072 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.592275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:37413 (local address 127.31.93.62:35693)
0809 04:54:41.592469 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:41.592475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:41.593709 (+ 1234us) server_negotiation.cc:407] Connection header received
0809 04:54:41.595472 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:41.595476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:41.595576 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:41.595656 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:41.596976 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.597715 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.599189 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.599584 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.599674 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.603452 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:41.603459 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:41.603464 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:41.603503 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:41.606669 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:41.607277 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:41.607280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:41.607283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:41.607404 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:41.607741 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:41.607744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:41.607745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:41.607992 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:41.608104 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:41.608679 (+ 575us) server_negotiation.cc:299] Negotiation successful
0809 04:54:41.608833 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 04:54:41.609884 29239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.592776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:35693 (local address 127.31.93.3:37413)
0809 04:54:41.593565 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:41.593617 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:41.595246 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:41.595828 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:41.595836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:41.596321 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:41.596837 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:41.596853 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.597831 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.597834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:41.599056 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:41.599077 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.599740 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.599744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:41.599924 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.600500 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:54:41.600531 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:41.603205 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:41.606786 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:41.606792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:41.606802 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:41.607188 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:41.607476 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:41.607486 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:41.607488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:41.607661 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:41.608082 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:41.608088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:41.608352 (+ 264us) client_negotiation.cc:769] Sending connection context
0809 04:54:41.608612 (+ 260us) client_negotiation.cc:240] Negotiation successful
0809 04:54:41.608945 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":325,"threads_started":1}
I20260809 04:54:41.611039 29237 heartbeater.cc:347] Connected to a master server at 127.31.93.62:35693
I20260809 04:54:41.611296 29237 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:41.611948 29237 heartbeater.cc:511] Master 127.31.93.62:35693 requested a full tablet report, sending...
I20260809 04:54:41.613198 28550 ts_manager.cc:194] Registered new tserver with Master: 027064c7b7ec47129b426bbe18272a7c (127.31.93.3:34797)
I20260809 04:54:41.613942 28550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:37413
I20260809 04:54:41.617660 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:54:41.630357 28842 heartbeater.cc:503] Master 127.31.93.62:35693 was elected leader, sending a full tablet report...
I20260809 04:54:41.633893 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.621598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.31.93.62:35693)
0809 04:54:41.621747 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:41.621751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:41.622182 (+ 431us) server_negotiation.cc:407] Connection header received
0809 04:54:41.622345 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:41.622350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:41.622412 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:41.622487 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:41.623899 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.624591 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.625855 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.626246 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.626378 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.628960 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:41.628966 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:41.628971 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:41.629010 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:41.631555 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:41.632313 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:41.632316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:41.632318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:41.632435 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:41.632770 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:41.632773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:41.632775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:41.633050 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:41.633229 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:41.633478 (+ 249us) server_negotiation.cc:299] Negotiation successful
0809 04:54:41.633599 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 04:54:41.637121 28550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:54:41.643783 28550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:54:41.665526 29257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.654875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.31.93.2:43189)
0809 04:54:41.657059 (+ 2184us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:41.657065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:41.657087 (+ 22us) server_negotiation.cc:407] Connection header received
0809 04:54:41.657159 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:41.657164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:41.657396 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:41.657564 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:41.658634 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.659883 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.662796 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.664205 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.664294 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.664447 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:41.665041 (+ 594us) server_negotiation.cc:299] Negotiation successful
0809 04:54:41.665198 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2031,"thread_start_us":1920,"threads_started":1}
I20260809 04:54:41.665616 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.654817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:43189 (local address 127.0.0.1:33964)
0809 04:54:41.656308 (+ 1491us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:41.656350 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:41.656536 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:41.657595 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:41.657603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:41.657632 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:41.658086 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:41.658104 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.660007 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.660013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:41.662433 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:41.662447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.664619 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.664624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:41.664826 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.664862 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 04:54:41.664973 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 04:54:41.665174 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"mutex_wait_us":35,"thread_start_us":5819,"threads_started":1}
I20260809 04:54:41.668398 29262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.655056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:41613 (local address 127.0.0.1:47262)
0809 04:54:41.660758 (+ 5702us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:41.660775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:41.660970 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:41.661523 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:41.661527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:41.661552 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:41.661935 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:41.661947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.663677 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.663683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:41.665549 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:41.665576 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.667626 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.667632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:41.667790 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.667824 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 04:54:41.667917 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 04:54:41.668036 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5542,"mutex_wait_us":26,"thread_start_us":828,"threads_started":1}
I20260809 04:54:41.668467 29261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.655693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.31.93.1:41613)
0809 04:54:41.660493 (+ 4800us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:41.660501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:41.660871 (+ 370us) server_negotiation.cc:407] Connection header received
0809 04:54:41.661123 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:41.661129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:41.661337 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:41.661518 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:41.662141 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.663558 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.665742 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.667169 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.667280 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.667483 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:41.667997 (+ 514us) server_negotiation.cc:299] Negotiation successful
0809 04:54:41.668160 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4633,"thread_start_us":4448,"threads_started":1}
I20260809 04:54:41.669463 29256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.658011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:34797 (local address 127.0.0.1:53590)
0809 04:54:41.661083 (+ 3072us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:41.661100 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:41.661225 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:41.661955 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:41.661959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:41.661978 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:41.662413 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:41.662427 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.664320 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.664325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:41.666769 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:41.666784 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.668745 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.668751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:41.668908 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.668933 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 04:54:41.669002 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 04:54:41.669128 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2910,"mutex_wait_us":17,"thread_start_us":2429,"threads_started":1}
I20260809 04:54:41.669322 28943 tablet_service.cc:1467] Processing CreateTablet for tablet 082c1d8fba3c43cebfd76374dafbcba6 (DEFAULT_TABLE table=test-table [id=436fe0afa6c6461485594b484fc485fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:41.670141 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.658741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53590 (local address 127.31.93.3:34797)
0809 04:54:41.659412 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:41.659420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:41.661195 (+ 1775us) server_negotiation.cc:407] Connection header received
0809 04:54:41.661365 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:41.661373 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:41.661581 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:41.661763 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:41.662584 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.664159 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.666954 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.668294 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.668412 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.668624 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:41.669701 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0809 04:54:41.669857 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":369,"threads_started":1}
I20260809 04:54:41.670841 28943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082c1d8fba3c43cebfd76374dafbcba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:41.672374 28742 tablet_service.cc:1467] Processing CreateTablet for tablet 082c1d8fba3c43cebfd76374dafbcba6 (DEFAULT_TABLE table=test-table [id=436fe0afa6c6461485594b484fc485fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:41.672734 29144 tablet_service.cc:1467] Processing CreateTablet for tablet 082c1d8fba3c43cebfd76374dafbcba6 (DEFAULT_TABLE table=test-table [id=436fe0afa6c6461485594b484fc485fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:41.673756 28742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082c1d8fba3c43cebfd76374dafbcba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:41.673882 29144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 082c1d8fba3c43cebfd76374dafbcba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:41.684324 29267 tablet_bootstrap.cc:492] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c: Bootstrap starting.
I20260809 04:54:41.687089 29268 tablet_bootstrap.cc:492] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411: Bootstrap starting.
I20260809 04:54:41.687861 29267 tablet_bootstrap.cc:654] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:41.688333 29269 tablet_bootstrap.cc:492] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5: Bootstrap starting.
I20260809 04:54:41.689445 29267 log.cc:824] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:41.690858 29268 tablet_bootstrap.cc:654] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:41.691561 29269 tablet_bootstrap.cc:654] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:41.692036 29268 log.cc:824] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:41.692464 29269 log.cc:824] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:41.692747 29267 tablet_bootstrap.cc:492] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c: No bootstrap required, opened a new log
I20260809 04:54:41.693004 29267 ts_tablet_manager.cc:1397] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 04:54:41.694880 29268 tablet_bootstrap.cc:492] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411: No bootstrap required, opened a new log
I20260809 04:54:41.695143 29268 ts_tablet_manager.cc:1397] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260809 04:54:41.695430 29269 tablet_bootstrap.cc:492] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5: No bootstrap required, opened a new log
I20260809 04:54:41.695678 29269 ts_tablet_manager.cc:1397] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 04:54:41.700255 29267 raft_consensus.cc:348] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70d7478178a4bf59123fb8480d32411" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43189 } } peers { permanent_uuid: "b02a0a02ed844e1da9b3bc6a17d4bab5" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41613 } } peers { permanent_uuid: "027064c7b7ec47129b426bbe18272a7c" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34797 } }
I20260809 04:54:41.700572 29267 raft_consensus.cc:374] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:41.700670 29267 raft_consensus.cc:729] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027064c7b7ec47129b426bbe18272a7c, State: Initialized, Role: FOLLOWER
I20260809 04:54:41.701189 29267 consensus_queue.cc:260] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70d7478178a4bf59123fb8480d32411" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43189 } } peers { permanent_uuid: "b02a0a02ed844e1da9b3bc6a17d4bab5" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41613 } } peers { permanent_uuid: "027064c7b7ec47129b426bbe18272a7c" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34797 } }
I20260809 04:54:41.702152 29237 heartbeater.cc:503] Master 127.31.93.62:35693 was elected leader, sending a full tablet report...
I20260809 04:54:41.702848 29267 ts_tablet_manager.cc:1428] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c: Time spent starting tablet: real 0.010s user 0.006s sys 0.006s
I20260809 04:54:41.705108 29268 raft_consensus.cc:348] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70d7478178a4bf59123fb8480d32411" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43189 } } peers { permanent_uuid: "b02a0a02ed844e1da9b3bc6a17d4bab5" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41613 } } peers { permanent_uuid: "027064c7b7ec47129b426bbe18272a7c" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34797 } }
I20260809 04:54:41.705574 29268 raft_consensus.cc:374] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:41.705657 29268 raft_consensus.cc:729] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e70d7478178a4bf59123fb8480d32411, State: Initialized, Role: FOLLOWER
I20260809 04:54:41.705529 29269 raft_consensus.cc:348] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70d7478178a4bf59123fb8480d32411" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43189 } } peers { permanent_uuid: "b02a0a02ed844e1da9b3bc6a17d4bab5" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41613 } } peers { permanent_uuid: "027064c7b7ec47129b426bbe18272a7c" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34797 } }
I20260809 04:54:41.705916 29269 raft_consensus.cc:374] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:41.706033 29269 raft_consensus.cc:729] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b02a0a02ed844e1da9b3bc6a17d4bab5, State: Initialized, Role: FOLLOWER
I20260809 04:54:41.706368 29268 consensus_queue.cc:260] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70d7478178a4bf59123fb8480d32411" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43189 } } peers { permanent_uuid: "b02a0a02ed844e1da9b3bc6a17d4bab5" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41613 } } peers { permanent_uuid: "027064c7b7ec47129b426bbe18272a7c" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34797 } }
I20260809 04:54:41.706617 29269 consensus_queue.cc:260] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70d7478178a4bf59123fb8480d32411" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43189 } } peers { permanent_uuid: "b02a0a02ed844e1da9b3bc6a17d4bab5" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41613 } } peers { permanent_uuid: "027064c7b7ec47129b426bbe18272a7c" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34797 } }
I20260809 04:54:41.707449 29033 heartbeater.cc:503] Master 127.31.93.62:35693 was elected leader, sending a full tablet report...
I20260809 04:54:41.708948 29269 ts_tablet_manager.cc:1428] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260809 04:54:41.710165 29268 ts_tablet_manager.cc:1428] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411: Time spent starting tablet: real 0.015s user 0.006s sys 0.007s
I20260809 04:54:41.759779 29280 raft_consensus.cc:482] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:54:41.760056 29280 raft_consensus.cc:504] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70d7478178a4bf59123fb8480d32411" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43189 } } peers { permanent_uuid: "b02a0a02ed844e1da9b3bc6a17d4bab5" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41613 } } peers { permanent_uuid: "027064c7b7ec47129b426bbe18272a7c" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34797 } }
I20260809 04:54:41.761699 29280 leader_election.cc:288] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b02a0a02ed844e1da9b3bc6a17d4bab5 (127.31.93.1:41613), 027064c7b7ec47129b426bbe18272a7c (127.31.93.3:34797)
I20260809 04:54:41.770257 29261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.762207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:36947 (local address 127.31.93.1:41613)
0809 04:54:41.762411 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:41.762417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:41.763166 (+ 749us) server_negotiation.cc:407] Connection header received
0809 04:54:41.763444 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:41.763449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:41.763525 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:41.763620 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:41.764274 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.765347 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.768091 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.769270 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.769399 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.769501 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:41.769889 (+ 388us) server_negotiation.cc:299] Negotiation successful
0809 04:54:41.770020 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:54:41.770336 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.762102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:43327 (local address 127.31.93.3:34797)
0809 04:54:41.762318 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:41.762323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:41.763228 (+ 905us) server_negotiation.cc:407] Connection header received
0809 04:54:41.763340 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:41.763345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:41.763423 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:41.763518 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:41.764217 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.765355 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.768091 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.769271 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.769399 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.769501 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:41.769892 (+ 391us) server_negotiation.cc:299] Negotiation successful
0809 04:54:41.770071 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 04:54:41.770503 29283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.762040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:34797 (local address 127.31.93.2:43327)
0809 04:54:41.763053 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:41.763068 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:41.763215 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:41.763573 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:41.763578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:41.763607 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:41.764038 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:41.764051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.765475 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.765481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:41.767953 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:41.767963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.769519 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.769524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:41.769757 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.769782 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 04:54:41.769850 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 04:54:41.770021 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":155,"thread_start_us":446,"threads_started":1}
I20260809 04:54:41.770581 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.762008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:41613 (local address 127.31.93.2:36947)
0809 04:54:41.763045 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:41.763081 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:41.763260 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:41.763696 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:41.763701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:41.763738 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:41.764141 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:41.764152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.765475 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.765481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:41.767941 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:41.767958 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.769496 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.769506 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:41.769710 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.769760 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 04:54:41.769856 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 04:54:41.770021 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":114,"spinlock_wait_cycles":39680,"thread_start_us":604,"threads_started":1}
I20260809 04:54:41.771483 29178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082c1d8fba3c43cebfd76374dafbcba6" candidate_uuid: "e70d7478178a4bf59123fb8480d32411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "027064c7b7ec47129b426bbe18272a7c" is_pre_election: true
I20260809 04:54:41.771701 28778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082c1d8fba3c43cebfd76374dafbcba6" candidate_uuid: "e70d7478178a4bf59123fb8480d32411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b02a0a02ed844e1da9b3bc6a17d4bab5" is_pre_election: true
I20260809 04:54:41.772116 29178 raft_consensus.cc:2454] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e70d7478178a4bf59123fb8480d32411 in term 0.
I20260809 04:54:41.772233 28778 raft_consensus.cc:2454] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e70d7478178a4bf59123fb8480d32411 in term 0.
I20260809 04:54:41.772828 28875 leader_election.cc:302] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 027064c7b7ec47129b426bbe18272a7c, e70d7478178a4bf59123fb8480d32411; no voters:
I20260809 04:54:41.773217 29280 raft_consensus.cc:2781] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:54:41.773351 29280 raft_consensus.cc:482] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:54:41.773442 29280 raft_consensus.cc:3037] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:41.775712 29280 raft_consensus.cc:504] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70d7478178a4bf59123fb8480d32411" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43189 } } peers { permanent_uuid: "b02a0a02ed844e1da9b3bc6a17d4bab5" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41613 } } peers { permanent_uuid: "027064c7b7ec47129b426bbe18272a7c" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34797 } }
I20260809 04:54:41.776269 29280 leader_election.cc:288] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [CANDIDATE]: Term 1 election: Requested vote from peers b02a0a02ed844e1da9b3bc6a17d4bab5 (127.31.93.1:41613), 027064c7b7ec47129b426bbe18272a7c (127.31.93.3:34797)
I20260809 04:54:41.776593 28778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082c1d8fba3c43cebfd76374dafbcba6" candidate_uuid: "e70d7478178a4bf59123fb8480d32411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b02a0a02ed844e1da9b3bc6a17d4bab5"
I20260809 04:54:41.776629 29178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "082c1d8fba3c43cebfd76374dafbcba6" candidate_uuid: "e70d7478178a4bf59123fb8480d32411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "027064c7b7ec47129b426bbe18272a7c"
I20260809 04:54:41.776754 28778 raft_consensus.cc:3037] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:41.776784 29178 raft_consensus.cc:3037] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:41.778880 28778 raft_consensus.cc:2454] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e70d7478178a4bf59123fb8480d32411 in term 1.
I20260809 04:54:41.778963 29178 raft_consensus.cc:2454] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e70d7478178a4bf59123fb8480d32411 in term 1.
I20260809 04:54:41.779261 28876 leader_election.cc:302] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b02a0a02ed844e1da9b3bc6a17d4bab5, e70d7478178a4bf59123fb8480d32411; no voters:
I20260809 04:54:41.779567 29280 raft_consensus.cc:2781] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:54:41.779973 29280 raft_consensus.cc:686] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [term 1 LEADER]: Becoming Leader. State: Replica: e70d7478178a4bf59123fb8480d32411, State: Running, Role: LEADER
I20260809 04:54:41.780448 29280 consensus_queue.cc:237] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70d7478178a4bf59123fb8480d32411" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43189 } } peers { permanent_uuid: "b02a0a02ed844e1da9b3bc6a17d4bab5" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41613 } } peers { permanent_uuid: "027064c7b7ec47129b426bbe18272a7c" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34797 } }
I20260809 04:54:41.784701 28549 catalog_manager.cc:5495] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 reported cstate change: term changed from 0 to 1, leader changed from <none> to e70d7478178a4bf59123fb8480d32411 (127.31.93.2). New cstate: current_term: 1 leader_uuid: "e70d7478178a4bf59123fb8480d32411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70d7478178a4bf59123fb8480d32411" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 43189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b02a0a02ed844e1da9b3bc6a17d4bab5" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "027064c7b7ec47129b426bbe18272a7c" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34797 } health_report { overall_health: UNKNOWN } } }
I20260809 04:54:41.829460 29257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.824130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.31.93.2:43189)
0809 04:54:41.824321 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:41.824328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:41.824395 (+ 67us) server_negotiation.cc:407] Connection header received
0809 04:54:41.824623 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:41.824627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:41.824706 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:41.824833 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:54:41.825218 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.825961 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.827390 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.827770 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.827878 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.828440 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:54:41.828891 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:54:41.829062 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:41.829196 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 04:54:41.829241 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:54:41.837188 28778 raft_consensus.cc:1261] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5 [term 1 FOLLOWER]: Refusing update from remote peer e70d7478178a4bf59123fb8480d32411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:54:41.837186 29178 raft_consensus.cc:1261] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c [term 1 FOLLOWER]: Refusing update from remote peer e70d7478178a4bf59123fb8480d32411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:54:41.838078 29284 consensus_queue.cc:1035] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "027064c7b7ec47129b426bbe18272a7c" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:54:41.838418 29280 consensus_queue.cc:1035] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b02a0a02ed844e1da9b3bc6a17d4bab5" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 04:54:41.841177 29238 tablet.cc:2369] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:54:41.862744 28550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:54:41.862987 28550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:54:41.866135 28550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eefa6c2fae7847bfb487c80324e20397: Sending DeleteTablet for 3 replicas of tablet 082c1d8fba3c43cebfd76374dafbcba6
I20260809 04:54:41.867141 28943 tablet_service.cc:1514] Processing DeleteTablet for tablet 082c1d8fba3c43cebfd76374dafbcba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:41 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:33964
I20260809 04:54:41.867198 28742 tablet_service.cc:1514] Processing DeleteTablet for tablet 082c1d8fba3c43cebfd76374dafbcba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:41 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:47262
I20260809 04:54:41.868100 29144 tablet_service.cc:1514] Processing DeleteTablet for tablet 082c1d8fba3c43cebfd76374dafbcba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:41 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:53590
I20260809 04:54:41.870110 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 28644
I20260809 04:54:41.872531 29303 tablet_replica.cc:330] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5: stopping tablet replica
I20260809 04:54:41.872944 29303 raft_consensus.cc:2229] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:54:41.873412 29303 raft_consensus.cc:2258] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:41.874207 29305 tablet_replica.cc:330] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411: stopping tablet replica
I20260809 04:54:41.874675 29305 raft_consensus.cc:2229] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:54:41.875360 29305 raft_consensus.cc:2258] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:41.877547 29303 ts_tablet_manager.cc:1905] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 04:54:41 dist-test-slave-4tdj krb5kdc[28476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:54:41.884958 29303 ts_tablet_manager.cc:1918] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:54:41.885126 29303 log.cc:1196] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/082c1d8fba3c43cebfd76374dafbcba6
I20260809 04:54:41.885690 29303 ts_tablet_manager.cc:1939] T 082c1d8fba3c43cebfd76374dafbcba6 P b02a0a02ed844e1da9b3bc6a17d4bab5: Deleting consensus metadata
I20260809 04:54:41.887615 29305 ts_tablet_manager.cc:1905] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:54:41.887740 28522 catalog_manager.cc:4841] TS b02a0a02ed844e1da9b3bc6a17d4bab5 (127.31.93.1:41613): tablet 082c1d8fba3c43cebfd76374dafbcba6 (table test-table [id=436fe0afa6c6461485594b484fc485fc]) successfully deleted
I20260809 04:54:41.888618 29306 tablet_replica.cc:330] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c: stopping tablet replica
I20260809 04:54:41.888960 29306 raft_consensus.cc:2229] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:54:41.889449 29306 raft_consensus.cc:2258] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:41.890635 29261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:41.871182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.31.93.1:41613)
0809 04:54:41.871343 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:41.871348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:41.871801 (+ 453us) server_negotiation.cc:407] Connection header received
0809 04:54:41.872060 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:41.872066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:41.872195 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:41.872311 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:41.874174 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.875328 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.877213 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:41.877862 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:41.877954 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:41.883311 (+ 5357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:41.883324 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:41.883340 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:41.883378 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:41.887367 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:41.888298 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:41.888307 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:41.888310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:41.888459 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:41.888902 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:41.888907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:41.888909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:41.889362 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:41.889657 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:41.890033 (+ 376us) server_negotiation.cc:299] Negotiation successful
0809 04:54:41.890243 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:54:41.891485 29306 ts_tablet_manager.cc:1905] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:54:41.892096 28834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:41.896580 29305 ts_tablet_manager.cc:1918] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:54:41.896718 29305 log.cc:1196] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/082c1d8fba3c43cebfd76374dafbcba6
I20260809 04:54:41.897167 29305 ts_tablet_manager.cc:1939] T 082c1d8fba3c43cebfd76374dafbcba6 P e70d7478178a4bf59123fb8480d32411: Deleting consensus metadata
I20260809 04:54:41.898252 29306 ts_tablet_manager.cc:1918] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:54:41.898454 29306 log.cc:1196] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/082c1d8fba3c43cebfd76374dafbcba6
I20260809 04:54:41.898553 28525 catalog_manager.cc:4841] TS e70d7478178a4bf59123fb8480d32411 (127.31.93.2:43189): tablet 082c1d8fba3c43cebfd76374dafbcba6 (table test-table [id=436fe0afa6c6461485594b484fc485fc]) successfully deleted
I20260809 04:54:41.898947 29306 ts_tablet_manager.cc:1939] T 082c1d8fba3c43cebfd76374dafbcba6 P 027064c7b7ec47129b426bbe18272a7c: Deleting consensus metadata
I20260809 04:54:41.900544 28521 catalog_manager.cc:4841] TS 027064c7b7ec47129b426bbe18272a7c (127.31.93.3:34797): tablet 082c1d8fba3c43cebfd76374dafbcba6 (table test-table [id=436fe0afa6c6461485594b484fc485fc]) successfully deleted
I20260809 04:54:41.982892 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 28644
I20260809 04:54:42.005539 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 28852
Aug 09 04:54:42 dist-test-slave-4tdj krb5kdc[28476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:54:42.022603 29257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:42.006446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33984 (local address 127.31.93.2:43189)
0809 04:54:42.006667 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:42.006674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:42.006697 (+ 23us) server_negotiation.cc:407] Connection header received
0809 04:54:42.006817 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:42.006823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:42.006906 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:42.007012 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:42.008188 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:42.009414 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:42.010694 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:42.011359 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:42.011470 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:42.015729 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:42.015747 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:42.015767 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:42.015811 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:42.019784 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:42.020477 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:42.020486 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:42.020490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:42.020653 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:42.021004 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:42.021009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:42.021013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:42.021447 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:42.021778 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:42.021959 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 04:54:42.022182 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 04:54:42.023399 29024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:42.141906 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 28852
I20260809 04:54:42.164505 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 29038
Aug 09 04:54:42 dist-test-slave-4tdj krb5kdc[28476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:54:42.179205 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:42.165252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.31.93.3:34797)
0809 04:54:42.165431 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:42.165435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:42.165454 (+ 19us) server_negotiation.cc:407] Connection header received
0809 04:54:42.165624 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:42.165627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:42.165693 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:42.165775 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:42.167152 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:42.167859 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:42.169127 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:42.169537 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:42.169647 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:42.173976 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:42.173992 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:42.174007 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:42.174048 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:42.176670 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:42.177261 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:42.177268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:42.177270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:42.177414 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:42.177684 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:42.177687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:42.177689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:42.178028 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:42.178238 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:42.178698 (+ 460us) server_negotiation.cc:299] Negotiation successful
0809 04:54:42.178878 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 04:54:42.179816 29230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:42.264854 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 29038
I20260809 04:54:42.285135 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 28500
I20260809 04:54:42.286185 28597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:42.393035 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 28500
2026-08-09T04:54:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4249 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-09T04:54:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:54:42Z Disabled control of system clock
I20260809 04:54:42.478098 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:44663
--webserver_interface=127.31.93.62
--webserver_port=0
--builtin_ntp_servers=127.31.93.20:35827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:44663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 04:54:42.799437 29320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:42.799738 29320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:42.799870 29320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:42.806911 29320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:54:42.807017 29320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:42.807175 29320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:42.807335 29320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:54:42.807402 29320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:54:42.816258 29320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:35827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:44663
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:44663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:42.817462 29320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:42.819247 29320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:42.830361 29329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:42.830448 29331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:42.830423 29327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:42.831295 29320 server_base.cc:1034] running on GCE node
I20260809 04:54:42.831995 29320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:42.833225 29320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:42.834426 29320 hybrid_clock.cc:648] HybridClock initialized: now 1786251282834369 us; error 39 us; skew 500 ppm
I20260809 04:54:42.834786 29320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:54:42.836876 29320 webserver.cc:460] Webserver started at http://127.31.93.62:39679/ using document root <none> and password file <none>
I20260809 04:54:42.837519 29320 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:42.837621 29320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:42.837914 29320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:42.840695 29320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "4fef32cb3bc04307b06efa66d6ef461e"
format_stamp: "Formatted at 2026-08-09 04:54:42 on dist-test-slave-4tdj"
server_key: "0256ccfeeda030a1ffcacc7c080f1dbf"
server_key_iv: "e081ffb7486c4e12585a384db960b5c3"
server_key_version: "encryptionkey@0"
I20260809 04:54:42.841455 29320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "4fef32cb3bc04307b06efa66d6ef461e"
format_stamp: "Formatted at 2026-08-09 04:54:42 on dist-test-slave-4tdj"
server_key: "0256ccfeeda030a1ffcacc7c080f1dbf"
server_key_iv: "e081ffb7486c4e12585a384db960b5c3"
server_key_version: "encryptionkey@0"
I20260809 04:54:42.846215 29320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:54:42.849407 29341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:42.850636 29320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:54:42.850790 29320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "4fef32cb3bc04307b06efa66d6ef461e"
format_stamp: "Formatted at 2026-08-09 04:54:42 on dist-test-slave-4tdj"
server_key: "0256ccfeeda030a1ffcacc7c080f1dbf"
server_key_iv: "e081ffb7486c4e12585a384db960b5c3"
server_key_version: "encryptionkey@0"
I20260809 04:54:42.850973 29320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:42.861793 29320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:42.862568 29320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:42.862820 29320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:42.879686 29320 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:44663
I20260809 04:54:42.879717 29416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:44663 every 8 connection(s)
I20260809 04:54:42.880856 29320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:54:42.884213 29417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:42.889315 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 29320
I20260809 04:54:42.889485 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:54:42.890031 32116 external_mini_cluster.cc:1428] Setting key 287ce6d4c78a1a8bd5e0e65622253795
I20260809 04:54:42.895115 29417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e: Bootstrap starting.
I20260809 04:54:42.897959 29417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:42.898947 29417 log.cc:824] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:42.901985 29417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e: No bootstrap required, opened a new log
I20260809 04:54:42.903014 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:42.891935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.31.93.62:44663)
0809 04:54:42.892938 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:42.892951 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:42.892992 (+ 41us) server_negotiation.cc:407] Connection header received
0809 04:54:42.894621 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:42.894644 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:42.895045 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:42.895401 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:42.895916 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:42.897180 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:42.898587 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:42.899258 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:42.899459 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:42.900449 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:42.900459 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:42.900476 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:42.901131 (+ 655us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:42.901245 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:42.901271 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:42.901390 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:42.901532 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 04:54:42.901813 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":359,"threads_started":1}
I20260809 04:54:42.909740 29417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fef32cb3bc04307b06efa66d6ef461e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44663 } }
I20260809 04:54:42.910043 29417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:42.910218 29417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fef32cb3bc04307b06efa66d6ef461e, State: Initialized, Role: FOLLOWER
I20260809 04:54:42.910974 29417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fef32cb3bc04307b06efa66d6ef461e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44663 } }
I20260809 04:54:42.911243 29417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:54:42.911375 29417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:54:42.911562 29417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:42.913581 29417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fef32cb3bc04307b06efa66d6ef461e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44663 } }
I20260809 04:54:42.914024 29417 leader_election.cc:302] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fef32cb3bc04307b06efa66d6ef461e; no voters:
I20260809 04:54:42.914608 29417 leader_election.cc:288] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:54:42.914722 29425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:54:42.916217 29417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:54:42.916599 29425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [term 1 LEADER]: Becoming Leader. State: Replica: 4fef32cb3bc04307b06efa66d6ef461e, State: Running, Role: LEADER
I20260809 04:54:42.917220 29425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fef32cb3bc04307b06efa66d6ef461e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44663 } }
I20260809 04:54:42.920238 29432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fef32cb3bc04307b06efa66d6ef461e. Latest consensus state: current_term: 1 leader_uuid: "4fef32cb3bc04307b06efa66d6ef461e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fef32cb3bc04307b06efa66d6ef461e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44663 } } }
I20260809 04:54:42.920665 29432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:42.921221 29431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fef32cb3bc04307b06efa66d6ef461e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fef32cb3bc04307b06efa66d6ef461e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44663 } } }
I20260809 04:54:42.921501 29431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:42.921870 29451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:54:42.926051 29451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:54:42.933645 29451 catalog_manager.cc:1347] Generated new cluster ID: cd36d6cf483640a8837cdbb8b953d748
I20260809 04:54:42.933754 29451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:54:42.952957 29451 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:54:42.954664 29451 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:54:42.967692 29451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e: Generated new TSK 0
I20260809 04:54:42.968592 29451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:54:42.985464 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44663
--builtin_ntp_servers=127.31.93.20:35827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 04:54:43.313580 29459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:43.313841 29459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:43.313964 29459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:43.320928 29459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:43.321022 29459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:43.321148 29459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:54:43.329665 29459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:35827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:43.330809 29459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:43.332445 29459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:43.344487 29467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:43.344436 29468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:43.344507 29471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:43.345124 29459 server_base.cc:1034] running on GCE node
I20260809 04:54:43.345690 29459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:43.346570 29459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:43.347779 29459 hybrid_clock.cc:648] HybridClock initialized: now 1786251283347722 us; error 43 us; skew 500 ppm
I20260809 04:54:43.348078 29459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:54:43.350190 29459 webserver.cc:460] Webserver started at http://127.31.93.1:43049/ using document root <none> and password file <none>
I20260809 04:54:43.350833 29459 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:43.350930 29459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:43.351213 29459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:43.353824 29459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "bdce8ae634d943e6b6f7d2dc3733d03d"
format_stamp: "Formatted at 2026-08-09 04:54:43 on dist-test-slave-4tdj"
server_key: "6f0f70ce59c596b7161e2513c4470142"
server_key_iv: "6b251e3722951ef160325f6d16454098"
server_key_version: "encryptionkey@0"
I20260809 04:54:43.354619 29459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "bdce8ae634d943e6b6f7d2dc3733d03d"
format_stamp: "Formatted at 2026-08-09 04:54:43 on dist-test-slave-4tdj"
server_key: "6f0f70ce59c596b7161e2513c4470142"
server_key_iv: "6b251e3722951ef160325f6d16454098"
server_key_version: "encryptionkey@0"
I20260809 04:54:43.359331 29459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:54:43.362505 29482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:43.363699 29459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:54:43.363847 29459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "bdce8ae634d943e6b6f7d2dc3733d03d"
format_stamp: "Formatted at 2026-08-09 04:54:43 on dist-test-slave-4tdj"
server_key: "6f0f70ce59c596b7161e2513c4470142"
server_key_iv: "6b251e3722951ef160325f6d16454098"
server_key_version: "encryptionkey@0"
I20260809 04:54:43.364028 29459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:43.387019 29459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:43.387696 29459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:43.387943 29459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:43.388823 29459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:43.390123 29459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:43.390216 29459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:43.390333 29459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:43.390415 29459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:43.419608 29459 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:41949
I20260809 04:54:43.419708 29670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:41949 every 8 connection(s)
I20260809 04:54:43.420814 29459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:54:43.423171 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 29459
I20260809 04:54:43.423338 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:54:43.423889 32116 external_mini_cluster.cc:1428] Setting key 45255ae473efbc9d3c340f39ee6d2b68
I20260809 04:54:43.426558 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44663
--builtin_ntp_servers=127.31.93.20:35827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 04:54:43.432809 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:43.423294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:51001 (local address 127.31.93.62:44663)
0809 04:54:43.423781 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:43.423787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:43.425011 (+ 1224us) server_negotiation.cc:407] Connection header received
0809 04:54:43.426901 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:43.426905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:43.426971 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:43.427063 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:43.428085 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:43.428769 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:43.430233 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:43.430670 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:43.430778 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:43.432021 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:43.432025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:43.432029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:43.432044 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:43.432086 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:43.432097 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:43.432169 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:43.432372 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 04:54:43.432465 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
I20260809 04:54:43.433701 29677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:43.423836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44663 (local address 127.31.93.1:51001)
0809 04:54:43.424852 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:43.424895 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:43.426532 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:43.427299 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:43.427308 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:43.427394 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:43.427925 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:43.427942 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:43.428904 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:43.428908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:43.430098 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:43.430111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:43.430892 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:43.430900 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:43.431118 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:43.431737 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:54:43.431740 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:43.431781 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:54:43.431855 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:43.432181 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:43.432192 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:43.432200 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 04:54:43.432346 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 04:54:43.432703 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":321,"threads_started":1}
I20260809 04:54:43.435207 29673 heartbeater.cc:347] Connected to a master server at 127.31.93.62:44663
I20260809 04:54:43.435528 29673 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:43.436262 29673 heartbeater.cc:511] Master 127.31.93.62:44663 requested a full tablet report, sending...
I20260809 04:54:43.438280 29366 ts_manager.cc:194] Registered new tserver with Master: bdce8ae634d943e6b6f7d2dc3733d03d (127.31.93.1:41949)
I20260809 04:54:43.439790 29366 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.93.1:51001
W20260809 04:54:43.759410 29678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:43.759711 29678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:43.759851 29678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:43.766938 29678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:43.767043 29678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:43.767198 29678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:54:43.776034 29678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:35827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:43.777220 29678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:43.779047 29678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:43.791718 29684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:43.791610 29685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:43.791621 29688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:43.792523 29678 server_base.cc:1034] running on GCE node
I20260809 04:54:43.793072 29678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:43.793920 29678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:43.795110 29678 hybrid_clock.cc:648] HybridClock initialized: now 1786251283795065 us; error 40 us; skew 500 ppm
I20260809 04:54:43.795395 29678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:54:43.797504 29678 webserver.cc:460] Webserver started at http://127.31.93.2:39797/ using document root <none> and password file <none>
I20260809 04:54:43.798135 29678 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:43.798233 29678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:43.798552 29678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:43.801170 29678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "a4bcfbaa7cd34def9501d9e051c28f8f"
format_stamp: "Formatted at 2026-08-09 04:54:43 on dist-test-slave-4tdj"
server_key: "ea5c0d6d110d10fc23c6ab4b04136fde"
server_key_iv: "41849741bff39c40631681ee86f6fb14"
server_key_version: "encryptionkey@0"
I20260809 04:54:43.801910 29678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "a4bcfbaa7cd34def9501d9e051c28f8f"
format_stamp: "Formatted at 2026-08-09 04:54:43 on dist-test-slave-4tdj"
server_key: "ea5c0d6d110d10fc23c6ab4b04136fde"
server_key_iv: "41849741bff39c40631681ee86f6fb14"
server_key_version: "encryptionkey@0"
I20260809 04:54:43.806639 29678 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260809 04:54:43.809808 29700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:43.811043 29678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:54:43.811187 29678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "a4bcfbaa7cd34def9501d9e051c28f8f"
format_stamp: "Formatted at 2026-08-09 04:54:43 on dist-test-slave-4tdj"
server_key: "ea5c0d6d110d10fc23c6ab4b04136fde"
server_key_iv: "41849741bff39c40631681ee86f6fb14"
server_key_version: "encryptionkey@0"
I20260809 04:54:43.811367 29678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:43.831813 29678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:43.832518 29678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:43.832770 29678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:43.833684 29678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:43.835052 29678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:43.835146 29678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:43.835247 29678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:43.835315 29678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:43.863878 29678 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:36891
I20260809 04:54:43.863942 29881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:36891 every 8 connection(s)
I20260809 04:54:43.865089 29678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:54:43.873147 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 29678
I20260809 04:54:43.873311 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:54:43.873754 32116 external_mini_cluster.cc:1428] Setting key c07627473b273ad609ec81612e3945f4
I20260809 04:54:43.876700 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44663
--builtin_ntp_servers=127.31.93.20:35827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 04:54:43.876914 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:43.867532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:52451 (local address 127.31.93.62:44663)
0809 04:54:43.867754 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:43.867761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:43.869185 (+ 1424us) server_negotiation.cc:407] Connection header received
0809 04:54:43.870873 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:43.870877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:43.870944 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:43.871032 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:43.871968 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:43.872674 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:43.874219 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:43.874712 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:43.874832 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:43.876174 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:43.876179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:43.876182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:43.876198 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:43.876231 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:43.876242 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:43.876309 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:43.876572 (+ 263us) server_negotiation.cc:299] Negotiation successful
0809 04:54:43.876647 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 04:54:43.878162 29888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:43.868148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44663 (local address 127.31.93.2:52451)
0809 04:54:43.869027 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:43.869070 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:43.870657 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:43.871220 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:43.871228 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:43.871308 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:43.871845 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:43.871861 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:43.872789 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:43.872792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:43.874075 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:43.874087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:43.874939 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:43.874947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:43.875170 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:43.875842 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:54:43.875845 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:43.875894 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:54:43.875998 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:43.876338 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:43.876349 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:43.876357 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 04:54:43.876564 (+ 207us) client_negotiation.cc:240] Negotiation successful
0809 04:54:43.876945 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":269,"threads_started":1}
I20260809 04:54:43.879634 29883 heartbeater.cc:347] Connected to a master server at 127.31.93.62:44663
I20260809 04:54:43.879932 29883 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:43.880753 29883 heartbeater.cc:511] Master 127.31.93.62:44663 requested a full tablet report, sending...
I20260809 04:54:43.882266 29366 ts_manager.cc:194] Registered new tserver with Master: a4bcfbaa7cd34def9501d9e051c28f8f (127.31.93.2:36891)
I20260809 04:54:43.883325 29366 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.93.2:52451
W20260809 04:54:44.196910 29889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:44.197162 29889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:44.197273 29889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:44.204156 29889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:44.204241 29889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:44.204347 29889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:54:44.212862 29889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:35827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:44663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:44.214056 29889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:44.215794 29889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:44.228266 29895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:44.228307 29900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:44.228307 29896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:44.229238 29889 server_base.cc:1034] running on GCE node
I20260809 04:54:44.229795 29889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:44.230718 29889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:44.231909 29889 hybrid_clock.cc:648] HybridClock initialized: now 1786251284231870 us; error 29 us; skew 500 ppm
I20260809 04:54:44.232180 29889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:54:44.234258 29889 webserver.cc:460] Webserver started at http://127.31.93.3:39933/ using document root <none> and password file <none>
I20260809 04:54:44.234910 29889 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:44.234997 29889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:44.235248 29889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:44.237962 29889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "96c6c9eb5c014c88821eb039faf385e9"
format_stamp: "Formatted at 2026-08-09 04:54:44 on dist-test-slave-4tdj"
server_key: "f3ae0526b67d0016606671fa7ef5c7a6"
server_key_iv: "a002be5cda379db8683c7cea42a8ada6"
server_key_version: "encryptionkey@0"
I20260809 04:54:44.238718 29889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "96c6c9eb5c014c88821eb039faf385e9"
format_stamp: "Formatted at 2026-08-09 04:54:44 on dist-test-slave-4tdj"
server_key: "f3ae0526b67d0016606671fa7ef5c7a6"
server_key_iv: "a002be5cda379db8683c7cea42a8ada6"
server_key_version: "encryptionkey@0"
I20260809 04:54:44.243353 29889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 04:54:44.246553 29906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:44.247751 29889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:54:44.247900 29889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "96c6c9eb5c014c88821eb039faf385e9"
format_stamp: "Formatted at 2026-08-09 04:54:44 on dist-test-slave-4tdj"
server_key: "f3ae0526b67d0016606671fa7ef5c7a6"
server_key_iv: "a002be5cda379db8683c7cea42a8ada6"
server_key_version: "encryptionkey@0"
I20260809 04:54:44.248082 29889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:44.266270 29889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:44.267076 29889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:44.267333 29889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:44.268277 29889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:44.269670 29889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:44.269757 29889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:44.269870 29889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:44.269939 29889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:44.299628 29889 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:36451
I20260809 04:54:44.299660 30096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:36451 every 8 connection(s)
I20260809 04:54:44.300850 29889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:54:44.307135 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 29889
I20260809 04:54:44.307313 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:54:44.307864 32116 external_mini_cluster.cc:1428] Setting key d9842f0c9c572a3c4a4c5bd054dfed8c
I20260809 04:54:44.313192 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.303374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:37369 (local address 127.31.93.62:44663)
0809 04:54:44.303553 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:44.303558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:44.304950 (+ 1392us) server_negotiation.cc:407] Connection header received
0809 04:54:44.306652 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:44.306656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:44.306724 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:44.306811 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:44.308184 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.308947 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.310591 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.311014 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.311133 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.312391 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:44.312396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:44.312399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:44.312416 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:44.312448 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:44.312459 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:44.312542 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:44.312746 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 04:54:44.312879 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:54:44.314096 30100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.303885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:44663 (local address 127.31.93.3:37369)
0809 04:54:44.304806 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:44.304842 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:44.306413 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:44.307106 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:44.307118 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:44.307232 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:44.308012 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:44.308038 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.309120 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.309125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:44.310450 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:44.310463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.311235 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.311242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:44.311459 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.312086 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:54:44.312088 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:44.312127 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:54:44.312202 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:44.312546 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:44.312557 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:44.312565 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 04:54:44.312722 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 04:54:44.313098 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":305,"threads_started":1}
I20260809 04:54:44.315315 30097 heartbeater.cc:347] Connected to a master server at 127.31.93.62:44663
I20260809 04:54:44.315575 30097 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:44.316293 30097 heartbeater.cc:511] Master 127.31.93.62:44663 requested a full tablet report, sending...
I20260809 04:54:44.317636 29366 ts_manager.cc:194] Registered new tserver with Master: 96c6c9eb5c014c88821eb039faf385e9 (127.31.93.3:36451)
I20260809 04:54:44.318480 29366 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.93.3:37369
I20260809 04:54:44.322520 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:54:44.331764 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.326476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.31.93.62:44663)
0809 04:54:44.326657 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:44.326664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:44.327063 (+ 399us) server_negotiation.cc:407] Connection header received
0809 04:54:44.327198 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:44.327201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:44.327259 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:44.327334 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:44.327760 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.328436 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.329659 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.330059 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.330140 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.331164 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:44.331169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:44.331172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:44.331187 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:44.331217 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:44.331229 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:44.331336 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:44.331451 (+ 115us) server_negotiation.cc:299] Negotiation successful
0809 04:54:44.331498 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 04:54:44.335253 29366 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:54:44.341830 29366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:54:44.365079 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.353748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:36451 (local address 127.0.0.1:42482)
0809 04:54:44.354846 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:44.354864 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:44.354970 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:44.357797 (+ 2827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:44.357804 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:44.357834 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:44.358410 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:44.358423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.360280 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.360285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:44.362891 (+ 2606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:44.362907 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.364168 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.364174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:44.364367 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.364422 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 04:54:44.364527 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 04:54:44.364697 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":56,"thread_start_us":2801,"threads_started":1}
I20260809 04:54:44.365477 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.353447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.31.93.2:36891)
0809 04:54:44.357498 (+ 4051us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:44.357506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:44.357535 (+ 29us) server_negotiation.cc:407] Connection header received
0809 04:54:44.357647 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:44.357654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:44.357874 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:44.358384 (+ 510us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:44.358887 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.360303 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.363081 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.364350 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.364516 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.364746 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:44.364924 (+ 178us) server_negotiation.cc:299] Negotiation successful
0809 04:54:44.365092 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3727,"thread_start_us":3600,"threads_started":1}
I20260809 04:54:44.365758 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.353200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:36891 (local address 127.0.0.1:52010)
0809 04:54:44.353898 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:44.353978 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:44.354154 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:44.358083 (+ 3929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:44.358087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:44.358110 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:44.358711 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:44.358721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.360452 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.360457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:44.362900 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:44.362909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.364590 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.364597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:44.364752 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.364784 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 04:54:44.364876 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 04:54:44.364989 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":67,"thread_start_us":311,"threads_started":1}
I20260809 04:54:44.366051 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.353939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.31.93.3:36451)
0809 04:54:44.357179 (+ 3240us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:44.357187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:44.357221 (+ 34us) server_negotiation.cc:407] Connection header received
0809 04:54:44.357310 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:44.357317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:44.357552 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:44.358899 (+ 1347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:44.359162 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.360158 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.363131 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.363997 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.364115 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.364387 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:44.365551 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0809 04:54:44.365707 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3065,"thread_start_us":2932,"threads_started":1}
I20260809 04:54:44.366508 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.354807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.31.93.1:41949)
0809 04:54:44.357642 (+ 2835us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:44.357650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:44.357677 (+ 27us) server_negotiation.cc:407] Connection header received
0809 04:54:44.357775 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:44.357782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:44.358213 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:44.358690 (+ 477us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:44.359251 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.360616 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.362971 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.364342 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.364473 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.364710 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:44.365980 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0809 04:54:44.366141 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2670,"thread_start_us":3471,"threads_started":1}
I20260809 04:54:44.366820 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.354281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:41949 (local address 127.0.0.1:42394)
0809 04:54:44.356771 (+ 2490us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:44.356790 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:44.356893 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:44.358704 (+ 1811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:44.358708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:44.358731 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:44.359077 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:44.359088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.360745 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.360749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:44.362779 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:44.362795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.364842 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.364848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:44.365038 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.365074 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 04:54:44.366344 (+ 1270us) client_negotiation.cc:240] Negotiation successful
0809 04:54:44.366467 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2365,"mutex_wait_us":60,"thread_start_us":828,"threads_started":1}
I20260809 04:54:44.369660 29984 tablet_service.cc:1467] Processing CreateTablet for tablet fd1d3c1cbf2d4b7595fd08d08b97620c (DEFAULT_TABLE table=test-table [id=fcc70152283040b6af1c5e6cfc8d5977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:44.369660 29779 tablet_service.cc:1467] Processing CreateTablet for tablet fd1d3c1cbf2d4b7595fd08d08b97620c (DEFAULT_TABLE table=test-table [id=fcc70152283040b6af1c5e6cfc8d5977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:44.371162 29779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1d3c1cbf2d4b7595fd08d08b97620c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:44.371160 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1d3c1cbf2d4b7595fd08d08b97620c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:44.373025 29560 tablet_service.cc:1467] Processing CreateTablet for tablet fd1d3c1cbf2d4b7595fd08d08b97620c (DEFAULT_TABLE table=test-table [id=fcc70152283040b6af1c5e6cfc8d5977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:44.374028 29560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1d3c1cbf2d4b7595fd08d08b97620c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:44.385178 30128 tablet_bootstrap.cc:492] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d: Bootstrap starting.
I20260809 04:54:44.387603 30130 tablet_bootstrap.cc:492] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9: Bootstrap starting.
I20260809 04:54:44.388971 30128 tablet_bootstrap.cc:654] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:44.390216 30128 log.cc:824] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:44.390781 30129 tablet_bootstrap.cc:492] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f: Bootstrap starting.
I20260809 04:54:44.390975 30130 tablet_bootstrap.cc:654] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:44.391999 30130 log.cc:824] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:44.393499 30128 tablet_bootstrap.cc:492] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d: No bootstrap required, opened a new log
I20260809 04:54:44.393736 30128 ts_tablet_manager.cc:1397] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 04:54:44.393810 30129 tablet_bootstrap.cc:654] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:44.394763 30129 log.cc:824] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:44.395648 30130 tablet_bootstrap.cc:492] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9: No bootstrap required, opened a new log
I20260809 04:54:44.395897 30130 ts_tablet_manager.cc:1397] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 04:54:44.396986 30129 tablet_bootstrap.cc:492] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f: No bootstrap required, opened a new log
I20260809 04:54:44.397212 30129 ts_tablet_manager.cc:1397] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260809 04:54:44.403710 30128 raft_consensus.cc:348] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcfbaa7cd34def9501d9e051c28f8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36891 } } peers { permanent_uuid: "bdce8ae634d943e6b6f7d2dc3733d03d" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41949 } } peers { permanent_uuid: "96c6c9eb5c014c88821eb039faf385e9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36451 } }
I20260809 04:54:44.404121 30128 raft_consensus.cc:374] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:44.403944 30129 raft_consensus.cc:348] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcfbaa7cd34def9501d9e051c28f8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36891 } } peers { permanent_uuid: "bdce8ae634d943e6b6f7d2dc3733d03d" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41949 } } peers { permanent_uuid: "96c6c9eb5c014c88821eb039faf385e9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36451 } }
I20260809 04:54:44.404217 30128 raft_consensus.cc:729] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdce8ae634d943e6b6f7d2dc3733d03d, State: Initialized, Role: FOLLOWER
I20260809 04:54:44.404296 30129 raft_consensus.cc:374] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:44.404407 30129 raft_consensus.cc:729] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bcfbaa7cd34def9501d9e051c28f8f, State: Initialized, Role: FOLLOWER
I20260809 04:54:44.404798 30128 consensus_queue.cc:260] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcfbaa7cd34def9501d9e051c28f8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36891 } } peers { permanent_uuid: "bdce8ae634d943e6b6f7d2dc3733d03d" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41949 } } peers { permanent_uuid: "96c6c9eb5c014c88821eb039faf385e9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36451 } }
I20260809 04:54:44.404982 30129 consensus_queue.cc:260] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcfbaa7cd34def9501d9e051c28f8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36891 } } peers { permanent_uuid: "bdce8ae634d943e6b6f7d2dc3733d03d" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41949 } } peers { permanent_uuid: "96c6c9eb5c014c88821eb039faf385e9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36451 } }
I20260809 04:54:44.405664 30130 raft_consensus.cc:348] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcfbaa7cd34def9501d9e051c28f8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36891 } } peers { permanent_uuid: "bdce8ae634d943e6b6f7d2dc3733d03d" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41949 } } peers { permanent_uuid: "96c6c9eb5c014c88821eb039faf385e9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36451 } }
I20260809 04:54:44.406016 29883 heartbeater.cc:503] Master 127.31.93.62:44663 was elected leader, sending a full tablet report...
I20260809 04:54:44.406072 30130 raft_consensus.cc:374] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:44.406165 30130 raft_consensus.cc:729] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96c6c9eb5c014c88821eb039faf385e9, State: Initialized, Role: FOLLOWER
I20260809 04:54:44.406657 30129 ts_tablet_manager.cc:1428] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f: Time spent starting tablet: real 0.009s user 0.010s sys 0.001s
I20260809 04:54:44.406869 30130 consensus_queue.cc:260] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcfbaa7cd34def9501d9e051c28f8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36891 } } peers { permanent_uuid: "bdce8ae634d943e6b6f7d2dc3733d03d" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41949 } } peers { permanent_uuid: "96c6c9eb5c014c88821eb039faf385e9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36451 } }
I20260809 04:54:44.407673 29673 heartbeater.cc:503] Master 127.31.93.62:44663 was elected leader, sending a full tablet report...
I20260809 04:54:44.408305 30128 ts_tablet_manager.cc:1428] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260809 04:54:44.409130 30097 heartbeater.cc:503] Master 127.31.93.62:44663 was elected leader, sending a full tablet report...
I20260809 04:54:44.409744 30130 ts_tablet_manager.cc:1428] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260809 04:54:44.422883 29674 tablet.cc:2369] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:54:44.552265 30140 raft_consensus.cc:482] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 04:54:44.552502 30099 tablet.cc:2369] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:54:44.552606 30140 raft_consensus.cc:504] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcfbaa7cd34def9501d9e051c28f8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36891 } } peers { permanent_uuid: "bdce8ae634d943e6b6f7d2dc3733d03d" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41949 } } peers { permanent_uuid: "96c6c9eb5c014c88821eb039faf385e9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36451 } }
I20260809 04:54:44.554505 30140 leader_election.cc:288] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4bcfbaa7cd34def9501d9e051c28f8f (127.31.93.2:36891), 96c6c9eb5c014c88821eb039faf385e9 (127.31.93.3:36451)
I20260809 04:54:44.560395 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.554868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:53229 (local address 127.31.93.2:36891)
0809 04:54:44.555084 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:44.555091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:44.555437 (+ 346us) server_negotiation.cc:407] Connection header received
0809 04:54:44.555624 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:44.555628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:44.555689 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:44.555761 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:44.556284 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.556972 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.558803 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.559565 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.559678 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.559773 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:44.560040 (+ 267us) server_negotiation.cc:299] Negotiation successful
0809 04:54:44.560163 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 04:54:44.560564 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.555162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:33633 (local address 127.31.93.3:36451)
0809 04:54:44.555341 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:44.555347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:44.555666 (+ 319us) server_negotiation.cc:407] Connection header received
0809 04:54:44.555832 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:44.555836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:44.555891 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:44.555960 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:44.556548 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.557223 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.558995 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.559737 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.559847 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.559934 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:44.560216 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 04:54:44.560334 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 04:54:44.560618 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.554750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:36451 (local address 127.31.93.1:33633)
0809 04:54:44.555561 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:44.555582 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:44.555711 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:44.556002 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:44.556006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:44.556042 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:44.556403 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:44.556412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.557357 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.557362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:44.558827 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:44.558840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.559925 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.559932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:44.560075 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.560098 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 04:54:44.560172 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 04:54:44.560282 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":21,"thread_start_us":591,"threads_started":1}
I20260809 04:54:44.560773 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.554680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:36891 (local address 127.31.93.1:53229)
0809 04:54:44.555333 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:44.555350 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:44.555489 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:44.555790 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:44.555795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:44.555820 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:44.556149 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:44.556160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.557110 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.557116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:44.558640 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:44.558652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.559733 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.559739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:44.559897 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.559923 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 04:54:44.560001 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 04:54:44.560132 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":1,"thread_start_us":370,"threads_started":1}
I20260809 04:54:44.561604 30017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1d3c1cbf2d4b7595fd08d08b97620c" candidate_uuid: "bdce8ae634d943e6b6f7d2dc3733d03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96c6c9eb5c014c88821eb039faf385e9" is_pre_election: true
I20260809 04:54:44.561658 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1d3c1cbf2d4b7595fd08d08b97620c" candidate_uuid: "bdce8ae634d943e6b6f7d2dc3733d03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bcfbaa7cd34def9501d9e051c28f8f" is_pre_election: true
I20260809 04:54:44.562031 30017 raft_consensus.cc:2454] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdce8ae634d943e6b6f7d2dc3733d03d in term 0.
I20260809 04:54:44.562095 29809 raft_consensus.cc:2454] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdce8ae634d943e6b6f7d2dc3733d03d in term 0.
I20260809 04:54:44.562669 29486 leader_election.cc:302] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96c6c9eb5c014c88821eb039faf385e9, bdce8ae634d943e6b6f7d2dc3733d03d; no voters:
I20260809 04:54:44.563011 30140 raft_consensus.cc:2781] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:54:44.563174 30140 raft_consensus.cc:482] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:54:44.563273 30140 raft_consensus.cc:3037] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:44.565281 30140 raft_consensus.cc:504] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcfbaa7cd34def9501d9e051c28f8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36891 } } peers { permanent_uuid: "bdce8ae634d943e6b6f7d2dc3733d03d" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41949 } } peers { permanent_uuid: "96c6c9eb5c014c88821eb039faf385e9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36451 } }
I20260809 04:54:44.565790 30140 leader_election.cc:288] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [CANDIDATE]: Term 1 election: Requested vote from peers a4bcfbaa7cd34def9501d9e051c28f8f (127.31.93.2:36891), 96c6c9eb5c014c88821eb039faf385e9 (127.31.93.3:36451)
I20260809 04:54:44.566179 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1d3c1cbf2d4b7595fd08d08b97620c" candidate_uuid: "bdce8ae634d943e6b6f7d2dc3733d03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bcfbaa7cd34def9501d9e051c28f8f"
I20260809 04:54:44.566247 30017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1d3c1cbf2d4b7595fd08d08b97620c" candidate_uuid: "bdce8ae634d943e6b6f7d2dc3733d03d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96c6c9eb5c014c88821eb039faf385e9"
I20260809 04:54:44.566398 29809 raft_consensus.cc:3037] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:44.566437 30017 raft_consensus.cc:3037] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:44.569535 29809 raft_consensus.cc:2454] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdce8ae634d943e6b6f7d2dc3733d03d in term 1.
I20260809 04:54:44.569523 30017 raft_consensus.cc:2454] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdce8ae634d943e6b6f7d2dc3733d03d in term 1.
I20260809 04:54:44.569900 29487 leader_election.cc:302] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bcfbaa7cd34def9501d9e051c28f8f, bdce8ae634d943e6b6f7d2dc3733d03d; no voters:
I20260809 04:54:44.570179 30140 raft_consensus.cc:2781] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:54:44.570619 30140 raft_consensus.cc:686] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [term 1 LEADER]: Becoming Leader. State: Replica: bdce8ae634d943e6b6f7d2dc3733d03d, State: Running, Role: LEADER
I20260809 04:54:44.571127 30140 consensus_queue.cc:237] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcfbaa7cd34def9501d9e051c28f8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36891 } } peers { permanent_uuid: "bdce8ae634d943e6b6f7d2dc3733d03d" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41949 } } peers { permanent_uuid: "96c6c9eb5c014c88821eb039faf385e9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36451 } }
I20260809 04:54:44.575645 29364 catalog_manager.cc:5495] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d reported cstate change: term changed from 0 to 1, leader changed from <none> to bdce8ae634d943e6b6f7d2dc3733d03d (127.31.93.1). New cstate: current_term: 1 leader_uuid: "bdce8ae634d943e6b6f7d2dc3733d03d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4bcfbaa7cd34def9501d9e051c28f8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdce8ae634d943e6b6f7d2dc3733d03d" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96c6c9eb5c014c88821eb039faf385e9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36451 } health_report { overall_health: UNKNOWN } } }
I20260809 04:54:44.597883 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.593099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.31.93.1:41949)
0809 04:54:44.593294 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:44.593298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:44.593360 (+ 62us) server_negotiation.cc:407] Connection header received
0809 04:54:44.593508 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:44.593511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:44.593600 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:44.593712 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:54:44.594005 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.594705 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.596045 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.596423 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.596507 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.596934 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:54:44.597378 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:54:44.597498 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:44.597591 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 04:54:44.597627 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:54:44.605368 29809 raft_consensus.cc:1261] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f [term 1 FOLLOWER]: Refusing update from remote peer bdce8ae634d943e6b6f7d2dc3733d03d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:54:44.605368 30017 raft_consensus.cc:1261] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9 [term 1 FOLLOWER]: Refusing update from remote peer bdce8ae634d943e6b6f7d2dc3733d03d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:54:44.606351 30149 consensus_queue.cc:1035] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bcfbaa7cd34def9501d9e051c28f8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:54:44.606676 30140 consensus_queue.cc:1035] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [LEADER]: Connected to new peer: Peer: permanent_uuid: "96c6c9eb5c014c88821eb039faf385e9" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:54:44.630213 29364 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:54:44.630479 29364 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:54:44.633754 29364 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fef32cb3bc04307b06efa66d6ef461e: Sending DeleteTablet for 3 replicas of tablet fd1d3c1cbf2d4b7595fd08d08b97620c
I20260809 04:54:44.634871 29779 tablet_service.cc:1514] Processing DeleteTablet for tablet fd1d3c1cbf2d4b7595fd08d08b97620c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:44 UTC) from {username='slave'} at 127.0.0.1:52010
I20260809 04:54:44.634855 29984 tablet_service.cc:1514] Processing DeleteTablet for tablet fd1d3c1cbf2d4b7595fd08d08b97620c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:44 UTC) from {username='slave'} at 127.0.0.1:42482
I20260809 04:54:44.635164 29560 tablet_service.cc:1514] Processing DeleteTablet for tablet fd1d3c1cbf2d4b7595fd08d08b97620c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:44 UTC) from {username='slave'} at 127.0.0.1:42394
I20260809 04:54:44.639226 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 29459
I20260809 04:54:44.647284 30169 tablet_replica.cc:330] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f: stopping tablet replica
I20260809 04:54:44.647531 30170 tablet_replica.cc:330] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d: stopping tablet replica
I20260809 04:54:44.647971 30170 raft_consensus.cc:2229] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:54:44.648497 30169 raft_consensus.cc:2229] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:54:44.648720 30170 raft_consensus.cc:2258] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:44.649545 30169 raft_consensus.cc:2258] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:44.649672 30166 tablet_replica.cc:330] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9: stopping tablet replica
I20260809 04:54:44.650051 30166 raft_consensus.cc:2229] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:54:44.650559 30166 raft_consensus.cc:2258] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:44.651822 30170 ts_tablet_manager.cc:1905] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:54:44.652261 30169 ts_tablet_manager.cc:1905] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:54:44.659310 30169 ts_tablet_manager.cc:1918] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:54:44.659467 30169 log.cc:1196] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/fd1d3c1cbf2d4b7595fd08d08b97620c
I20260809 04:54:44.659979 30170 ts_tablet_manager.cc:1918] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:54:44.660055 30169 ts_tablet_manager.cc:1939] T fd1d3c1cbf2d4b7595fd08d08b97620c P a4bcfbaa7cd34def9501d9e051c28f8f: Deleting consensus metadata
I20260809 04:54:44.660163 30170 log.cc:1196] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/fd1d3c1cbf2d4b7595fd08d08b97620c
I20260809 04:54:44.660765 30170 ts_tablet_manager.cc:1939] T fd1d3c1cbf2d4b7595fd08d08b97620c P bdce8ae634d943e6b6f7d2dc3733d03d: Deleting consensus metadata
I20260809 04:54:44.662041 29346 catalog_manager.cc:4841] TS a4bcfbaa7cd34def9501d9e051c28f8f (127.31.93.2:36891): tablet fd1d3c1cbf2d4b7595fd08d08b97620c (table test-table [id=fcc70152283040b6af1c5e6cfc8d5977]) successfully deleted
I20260809 04:54:44.662686 29346 catalog_manager.cc:4841] TS bdce8ae634d943e6b6f7d2dc3733d03d (127.31.93.1:41949): tablet fd1d3c1cbf2d4b7595fd08d08b97620c (table test-table [id=fcc70152283040b6af1c5e6cfc8d5977]) successfully deleted
I20260809 04:54:44.663563 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.641276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.31.93.1:41949)
0809 04:54:44.642011 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:44.642020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:44.654983 (+ 12963us) server_negotiation.cc:407] Connection header received
0809 04:54:44.655383 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:44.655389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:44.655483 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:44.655576 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:44.656297 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.657526 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.659414 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.660097 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.660194 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.661834 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:44.661845 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:44.661863 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:44.662691 (+ 828us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:44.662752 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:44.662777 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:44.662864 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:44.663069 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 04:54:44.663171 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598}
I20260809 04:54:44.663645 30166 ts_tablet_manager.cc:1905] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:54:44.664258 29666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:44.669585 30166 ts_tablet_manager.cc:1918] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:54:44.669708 30166 log.cc:1196] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/fd1d3c1cbf2d4b7595fd08d08b97620c
I20260809 04:54:44.670111 30166 ts_tablet_manager.cc:1939] T fd1d3c1cbf2d4b7595fd08d08b97620c P 96c6c9eb5c014c88821eb039faf385e9: Deleting consensus metadata
I20260809 04:54:44.671476 29345 catalog_manager.cc:4841] TS 96c6c9eb5c014c88821eb039faf385e9 (127.31.93.3:36451): tablet fd1d3c1cbf2d4b7595fd08d08b97620c (table test-table [id=fcc70152283040b6af1c5e6cfc8d5977]) successfully deleted
I20260809 04:54:44.760818 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 29459
I20260809 04:54:44.782470 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 29678
I20260809 04:54:44.789044 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.783203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52018 (local address 127.31.93.2:36891)
0809 04:54:44.783410 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:44.783414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:44.783433 (+ 19us) server_negotiation.cc:407] Connection header received
0809 04:54:44.783623 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:44.783627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:44.783723 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:44.783806 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:44.784150 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.784858 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.786119 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.786581 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.786673 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.787829 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:44.787840 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:44.787855 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:44.788436 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:44.788484 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:44.788522 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:44.788613 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:44.788703 (+ 90us) server_negotiation.cc:299] Negotiation successful
0809 04:54:44.788794 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 04:54:44.789664 29876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:44.875814 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 29678
I20260809 04:54:44.904994 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 29889
I20260809 04:54:44.914477 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:44.907207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.31.93.3:36451)
0809 04:54:44.907415 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:44.907423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:44.907565 (+ 142us) server_negotiation.cc:407] Connection header received
0809 04:54:44.907754 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:44.907760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:44.907870 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:44.907977 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:44.908433 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.909596 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.911169 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:44.911779 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:44.911898 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:44.913074 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:44.913086 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:44.913105 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:44.913795 (+ 690us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:44.913848 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:44.913870 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:44.913963 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:44.914127 (+ 164us) server_negotiation.cc:299] Negotiation successful
0809 04:54:44.914217 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 04:54:44.916662 30086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:45.011610 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 29889
I20260809 04:54:45.033634 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 29320
I20260809 04:54:45.034703 29411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:45.140847 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 29320
2026-08-09T04:54:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2746 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:54:45 dist-test-slave-4tdj krb5kdc[30181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:54:45 dist-test-slave-4tdj krb5kdc[30181](info): set up 2 sockets
Aug 09 04:54:45 dist-test-slave-4tdj krb5kdc[30181](info): commencing operation
krb5kdc: starting...
W20260809 04:54:46.360208 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:54:46 dist-test-slave-4tdj krb5kdc[30181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:54:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:54:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:54:46.603089 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:34939
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:34939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:46.918814 30201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:46.919072 30201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:46.919201 30201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:46.926038 30201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:54:46.926134 30201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:46.926205 30201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:46.926281 30201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:54:46.926357 30201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:54:46.935496 30201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:34939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:34939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:46.936605 30201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:46.937805 30201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:46.948515 30214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:46.948542 30216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:46.948886 30219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:46.949290 30201 server_base.cc:1034] running on GCE node
I20260809 04:54:46.950011 30201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:46.951313 30201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:46.952512 30201 hybrid_clock.cc:648] HybridClock initialized: now 1786251286952455 us; error 39 us; skew 500 ppm
I20260809 04:54:46.952836 30201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:46 dist-test-slave-4tdj krb5kdc[30181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251286, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:46.958177 30201 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:54:46.959520 30201 webserver.cc:460] Webserver started at http://127.31.93.62:37491/ using document root <none> and password file <none>
I20260809 04:54:46.960119 30201 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:46.960212 30201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:46.960443 30201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:46.963267 30201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "8b7cda3236cb48529b69b28df6676da1"
format_stamp: "Formatted at 2026-08-09 04:54:46 on dist-test-slave-4tdj"
server_key: "9c77c79200cbd83b72f15cd1a77755a6"
server_key_iv: "6dc51eb3948919ecb82b374c0b21f4f0"
server_key_version: "encryptionkey@0"
I20260809 04:54:46.963932 30201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "8b7cda3236cb48529b69b28df6676da1"
format_stamp: "Formatted at 2026-08-09 04:54:46 on dist-test-slave-4tdj"
server_key: "9c77c79200cbd83b72f15cd1a77755a6"
server_key_iv: "6dc51eb3948919ecb82b374c0b21f4f0"
server_key_version: "encryptionkey@0"
I20260809 04:54:46.968690 30201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:54:46.971868 30229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:46.973047 30201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:54:46.973191 30201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "8b7cda3236cb48529b69b28df6676da1"
format_stamp: "Formatted at 2026-08-09 04:54:46 on dist-test-slave-4tdj"
server_key: "9c77c79200cbd83b72f15cd1a77755a6"
server_key_iv: "6dc51eb3948919ecb82b374c0b21f4f0"
server_key_version: "encryptionkey@0"
I20260809 04:54:46.973337 30201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:46.991756 30201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:46.995265 30201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:46.995478 30201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:47.012460 30201 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:34939
I20260809 04:54:47.012521 30315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:34939 every 8 connection(s)
I20260809 04:54:47.013664 30201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:54:47.017145 30317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:47.023447 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 30201
I20260809 04:54:47.023633 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:54:47.024152 32116 external_mini_cluster.cc:1428] Setting key b65dedb82ae1f21158db76fb8d5d7f8c
I20260809 04:54:47.031629 30317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1: Bootstrap starting.
I20260809 04:54:47.035684 30317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:47.036878 30317 log.cc:824] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:47.039541 30317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1: No bootstrap required, opened a new log
Aug 09 04:54:47 dist-test-slave-4tdj krb5kdc[30181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:54:47.046999 30317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7cda3236cb48529b69b28df6676da1" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34939 } }
I20260809 04:54:47.047418 30317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:47.047530 30317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7cda3236cb48529b69b28df6676da1, State: Initialized, Role: FOLLOWER
I20260809 04:54:47.048139 30317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7cda3236cb48529b69b28df6676da1" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34939 } }
I20260809 04:54:47.048434 30317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:54:47.048583 30317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:54:47.048755 30317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:47.050925 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:47.030630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.31.93.62:34939)
0809 04:54:47.031460 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:47.031474 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:47.031520 (+ 46us) server_negotiation.cc:407] Connection header received
0809 04:54:47.033866 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:47.033898 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:47.034396 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:47.034792 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:47.036249 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:47.037643 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:47.039006 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:47.039690 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:47.039845 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:47.043878 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:47.043886 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:47.043899 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:47.043939 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:47.046945 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:47.047731 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:47.047738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:47.047743 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:47.047907 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:47.048331 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:47.048334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:47.048336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:47.048932 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:47.049160 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:47.049592 (+ 432us) server_negotiation.cc:299] Negotiation successful
0809 04:54:47.049932 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":458,"threads_started":1}
I20260809 04:54:47.051334 30317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7cda3236cb48529b69b28df6676da1" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34939 } }
I20260809 04:54:47.051791 30317 leader_election.cc:302] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b7cda3236cb48529b69b28df6676da1; no voters:
I20260809 04:54:47.052356 30317 leader_election.cc:288] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:54:47.052421 30327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:54:47.054283 30317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:54:47.054567 30327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7cda3236cb48529b69b28df6676da1, State: Running, Role: LEADER
I20260809 04:54:47.055069 30327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7cda3236cb48529b69b28df6676da1" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34939 } }
I20260809 04:54:47.057870 30329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7cda3236cb48529b69b28df6676da1. Latest consensus state: current_term: 1 leader_uuid: "8b7cda3236cb48529b69b28df6676da1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7cda3236cb48529b69b28df6676da1" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34939 } } }
I20260809 04:54:47.058266 30329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:47.060683 30328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b7cda3236cb48529b69b28df6676da1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7cda3236cb48529b69b28df6676da1" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 34939 } } }
I20260809 04:54:47.061028 30328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1 [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:47.063119 30342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:54:47.067217 30342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:54:47.074921 30342 catalog_manager.cc:1347] Generated new cluster ID: fbc91a8eafab42fca0c87c28c68af2be
I20260809 04:54:47.075024 30342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:54:47.092172 30342 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:54:47.093324 30342 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:54:47.112391 30342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1: Generated new TSK 0
I20260809 04:54:47.113091 30342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.93.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
Entry for principal kudu/127.31.93.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab.
I20260809 04:54:47.179229 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34939
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:47.518041 30365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:47.518286 30365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:47.518415 30365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:47.525238 30365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:47.525331 30365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:47.525439 30365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:54:47.534526 30365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.1.keytab
--principal=kudu/127.31.93.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:47.535596 30365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:47.536760 30365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:47.548663 30370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:47.548691 30374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:47.548774 30371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:47.549814 30365 server_base.cc:1034] running on GCE node
I20260809 04:54:47.550405 30365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:47.551247 30365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:47.552444 30365 hybrid_clock.cc:648] HybridClock initialized: now 1786251287552392 us; error 52 us; skew 500 ppm
I20260809 04:54:47.552726 30365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:47 dist-test-slave-4tdj krb5kdc[30181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251287, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:47.555920 30365 init.cc:377] Logged in from keytab as kudu/127.31.93.1@KRBTEST.COM (short username kudu)
I20260809 04:54:47.557686 30365 webserver.cc:460] Webserver started at http://127.31.93.1:45523/ using document root <none> and password file <none>
I20260809 04:54:47.558393 30365 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:47.558499 30365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:47.558758 30365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:47.561525 30365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "c5bf11bd406a40e1913a51bcdd6fb376"
format_stamp: "Formatted at 2026-08-09 04:54:47 on dist-test-slave-4tdj"
server_key: "cc8179d80613514da69ffbe88433be97"
server_key_iv: "1da4f07c6509c9a25dd2e7797ce54686"
server_key_version: "encryptionkey@0"
I20260809 04:54:47.562417 30365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "c5bf11bd406a40e1913a51bcdd6fb376"
format_stamp: "Formatted at 2026-08-09 04:54:47 on dist-test-slave-4tdj"
server_key: "cc8179d80613514da69ffbe88433be97"
server_key_iv: "1da4f07c6509c9a25dd2e7797ce54686"
server_key_version: "encryptionkey@0"
I20260809 04:54:47.567304 30365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 04:54:47.570560 30383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:47.571745 30365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 04:54:47.571888 30365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "c5bf11bd406a40e1913a51bcdd6fb376"
format_stamp: "Formatted at 2026-08-09 04:54:47 on dist-test-slave-4tdj"
server_key: "cc8179d80613514da69ffbe88433be97"
server_key_iv: "1da4f07c6509c9a25dd2e7797ce54686"
server_key_version: "encryptionkey@0"
I20260809 04:54:47.572054 30365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:47.603217 30365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:47.606788 30365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:47.607036 30365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:47.607919 30365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:47.609171 30365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:47.609241 30365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:47.609361 30365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:47.609412 30365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:47.639838 30365 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:44077
I20260809 04:54:47.639902 30590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:44077 every 8 connection(s)
I20260809 04:54:47.641031 30365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:54:47.644932 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 30365
I20260809 04:54:47.645118 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:54:47.645568 32116 external_mini_cluster.cc:1428] Setting key e6ab53f22c397b678cb5d1c2ae1994bd
Aug 09 04:54:47 dist-test-slave-4tdj krb5kdc[30181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251287, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.1@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:54:47.659520 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:47.643438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:58843 (local address 127.31.93.62:34939)
0809 04:54:47.643845 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:47.643850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:47.644785 (+ 935us) server_negotiation.cc:407] Connection header received
0809 04:54:47.646680 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:47.646686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:47.646768 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:47.646864 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:47.648273 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:47.648965 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:47.650496 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:47.650881 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:47.650990 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:47.654199 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:47.654204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:47.654207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:47.654239 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:47.656654 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:47.657312 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:47.657318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:47.657322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:47.657444 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:47.657849 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:47.657854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:47.657857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:47.658122 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:47.658254 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:47.658950 (+ 696us) server_negotiation.cc:299] Negotiation successful
0809 04:54:47.659179 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":188,"threads_started":1}
I20260809 04:54:47.660440 30595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:47.643947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:34939 (local address 127.31.93.1:58843)
0809 04:54:47.644635 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:47.644680 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:47.646421 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:47.647096 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:47.647105 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:47.647574 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:47.648111 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:47.648129 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:47.649126 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:47.649130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:47.650357 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:47.650369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:47.651103 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:47.651111 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:47.651330 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:47.651927 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:54:47.651951 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:47.653992 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:47.656784 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:47.656793 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:47.656807 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:47.657199 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:47.657546 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:47.657551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:47.657555 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:47.657742 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:47.658247 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:47.658255 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:47.658596 (+ 341us) client_negotiation.cc:769] Sending connection context
0809 04:54:47.658838 (+ 242us) client_negotiation.cc:240] Negotiation successful
0809 04:54:47.659322 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":270,"threads_started":1}
I20260809 04:54:47.661923 30591 heartbeater.cc:347] Connected to a master server at 127.31.93.62:34939
I20260809 04:54:47.662257 30591 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:47.663097 30591 heartbeater.cc:511] Master 127.31.93.62:34939 requested a full tablet report, sending...
I20260809 04:54:47.665447 30261 ts_manager.cc:194] Registered new tserver with Master: c5bf11bd406a40e1913a51bcdd6fb376 (127.31.93.1:44077)
I20260809 04:54:47.667403 30261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.1@KRBTEST.COM'} at 127.31.93.1:58843
WARNING: no policy specified for kudu/127.31.93.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
Entry for principal kudu/127.31.93.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab.
I20260809 04:54:47.705094 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34939
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:48.033226 30601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:48.033597 30601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:48.033758 30601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:48.041257 30601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:48.041356 30601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:48.041486 30601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:54:48.050788 30601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.2.keytab
--principal=kudu/127.31.93.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:48.051864 30601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:48.053083 30601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:48.065630 30610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:48.065601 30613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:48.065675 30611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:48.066726 30601 server_base.cc:1034] running on GCE node
I20260809 04:54:48.067287 30601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:48.068150 30601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:48.069350 30601 hybrid_clock.cc:648] HybridClock initialized: now 1786251288069300 us; error 37 us; skew 500 ppm
I20260809 04:54:48.069653 30601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:48 dist-test-slave-4tdj krb5kdc[30181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251288, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:48.072640 30601 init.cc:377] Logged in from keytab as kudu/127.31.93.2@KRBTEST.COM (short username kudu)
I20260809 04:54:48.073928 30601 webserver.cc:460] Webserver started at http://127.31.93.2:36625/ using document root <none> and password file <none>
I20260809 04:54:48.074630 30601 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:48.074759 30601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:48.075043 30601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:48.077922 30601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "72626b15e1de4418a5a80f522e3512d0"
format_stamp: "Formatted at 2026-08-09 04:54:48 on dist-test-slave-4tdj"
server_key: "56d803830b28963b1827f45db4adc485"
server_key_iv: "8252b951a952421d6f238167c4d33d95"
server_key_version: "encryptionkey@0"
I20260809 04:54:48.078680 30601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "72626b15e1de4418a5a80f522e3512d0"
format_stamp: "Formatted at 2026-08-09 04:54:48 on dist-test-slave-4tdj"
server_key: "56d803830b28963b1827f45db4adc485"
server_key_iv: "8252b951a952421d6f238167c4d33d95"
server_key_version: "encryptionkey@0"
I20260809 04:54:48.083379 30601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 04:54:48.086508 30627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:48.087738 30601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:54:48.087872 30601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "72626b15e1de4418a5a80f522e3512d0"
format_stamp: "Formatted at 2026-08-09 04:54:48 on dist-test-slave-4tdj"
server_key: "56d803830b28963b1827f45db4adc485"
server_key_iv: "8252b951a952421d6f238167c4d33d95"
server_key_version: "encryptionkey@0"
I20260809 04:54:48.088043 30601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:48.098762 30601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:48.102654 30601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:48.102901 30601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:48.103760 30601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:48.105026 30601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:48.105106 30601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:48.105232 30601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:48.105283 30601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:48.135030 30601 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:35049
I20260809 04:54:48.135128 30819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:35049 every 8 connection(s)
I20260809 04:54:48.136268 30601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:54:48.146101 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 30601
I20260809 04:54:48.146279 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:54:48.146765 32116 external_mini_cluster.cc:1428] Setting key 7cf229a92102bc11320dde779e87eeaf
Aug 09 04:54:48 dist-test-slave-4tdj krb5kdc[30181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251288, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.2@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:54:48.155210 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:48.138685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:32821 (local address 127.31.93.62:34939)
0809 04:54:48.138845 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:48.138850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:48.139954 (+ 1104us) server_negotiation.cc:407] Connection header received
0809 04:54:48.141728 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:48.141734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:48.141819 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:48.141909 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:48.143474 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.144236 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.145782 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.146220 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.146365 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:48.149730 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:48.149735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:48.149738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:48.149773 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:48.152277 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:48.152937 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:48.152942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:48.152944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:48.153075 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:48.153471 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:48.153475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:48.153477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:48.153753 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:48.153888 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:48.154635 (+ 747us) server_negotiation.cc:299] Negotiation successful
0809 04:54:48.154869 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:54:48.156132 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:48.139117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:34939 (local address 127.31.93.2:32821)
0809 04:54:48.139795 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:48.139834 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:48.141455 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:48.142148 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:48.142156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:48.142738 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:48.143296 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:48.143314 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.144378 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.144382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:48.145630 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:48.145648 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.146447 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.146454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:48.146675 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:48.147288 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:54:48.147318 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:48.149517 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:48.152407 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:48.152416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:48.152430 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:48.152824 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:48.153181 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:48.153185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:48.153189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:48.153364 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:48.153879 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:48.153886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:48.154215 (+ 329us) client_negotiation.cc:769] Sending connection context
0809 04:54:48.154511 (+ 296us) client_negotiation.cc:240] Negotiation successful
0809 04:54:48.155016 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":254,"threads_started":1}
I20260809 04:54:48.157701 30825 heartbeater.cc:347] Connected to a master server at 127.31.93.62:34939
I20260809 04:54:48.157984 30825 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:48.158739 30825 heartbeater.cc:511] Master 127.31.93.62:34939 requested a full tablet report, sending...
I20260809 04:54:48.160159 30261 ts_manager.cc:194] Registered new tserver with Master: 72626b15e1de4418a5a80f522e3512d0 (127.31.93.2:35049)
I20260809 04:54:48.161067 30261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.2@KRBTEST.COM'} at 127.31.93.2:32821
WARNING: no policy specified for kudu/127.31.93.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
Entry for principal kudu/127.31.93.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab.
I20260809 04:54:48.203214 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34939
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:48.515524 30833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:48.515815 30833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:48.515959 30833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:48.522989 30833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:48.523106 30833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:48.523264 30833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:54:48.532706 30833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.3.keytab
--principal=kudu/127.31.93.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:34939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:48.533962 30833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:48.535306 30833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:48.547748 30845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:48.547953 30848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:48.548198 30843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:48.548563 30833 server_base.cc:1034] running on GCE node
I20260809 04:54:48.549201 30833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:48.550138 30833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:48.551332 30833 hybrid_clock.cc:648] HybridClock initialized: now 1786251288551288 us; error 42 us; skew 500 ppm
I20260809 04:54:48.551628 30833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:48 dist-test-slave-4tdj krb5kdc[30181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251288, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:48.554596 30833 init.cc:377] Logged in from keytab as kudu/127.31.93.3@KRBTEST.COM (short username kudu)
I20260809 04:54:48.555946 30833 webserver.cc:460] Webserver started at http://127.31.93.3:45011/ using document root <none> and password file <none>
I20260809 04:54:48.556566 30833 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:48.556663 30833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:48.556917 30833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:48.559772 30833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "1b7efb73f2114b88869101753a51c8c1"
format_stamp: "Formatted at 2026-08-09 04:54:48 on dist-test-slave-4tdj"
server_key: "58821221e98618c6df2b23310fa4c0d9"
server_key_iv: "91fe5cb042cce6f518e483e59aa7d979"
server_key_version: "encryptionkey@0"
I20260809 04:54:48.560508 30833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "1b7efb73f2114b88869101753a51c8c1"
format_stamp: "Formatted at 2026-08-09 04:54:48 on dist-test-slave-4tdj"
server_key: "58821221e98618c6df2b23310fa4c0d9"
server_key_iv: "91fe5cb042cce6f518e483e59aa7d979"
server_key_version: "encryptionkey@0"
I20260809 04:54:48.565313 30833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:54:48.568611 30863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:48.569834 30833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:54:48.569978 30833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "1b7efb73f2114b88869101753a51c8c1"
format_stamp: "Formatted at 2026-08-09 04:54:48 on dist-test-slave-4tdj"
server_key: "58821221e98618c6df2b23310fa4c0d9"
server_key_iv: "91fe5cb042cce6f518e483e59aa7d979"
server_key_version: "encryptionkey@0"
I20260809 04:54:48.570191 30833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:48.595965 30833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:48.599767 30833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:48.600026 30833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:48.601043 30833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:48.602535 30833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:48.602612 30833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:48.602725 30833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:48.602775 30833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:48.632311 30833 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:33409
I20260809 04:54:48.632357 31064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:33409 every 8 connection(s)
I20260809 04:54:48.633526 30833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:54:48.633955 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 30833
I20260809 04:54:48.634143 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:54:48.634692 32116 external_mini_cluster.cc:1428] Setting key 72a8380bc3ac32ecf501091b258eeaf3
Aug 09 04:54:48 dist-test-slave-4tdj krb5kdc[30181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251288, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.3@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:54:48.652434 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:48.636142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:38069 (local address 127.31.93.62:34939)
0809 04:54:48.636341 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:48.636346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:48.637709 (+ 1363us) server_negotiation.cc:407] Connection header received
0809 04:54:48.639549 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:48.639562 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:48.639649 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:48.639744 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:48.641110 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.642219 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.643687 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.644288 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.644421 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:48.647496 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:48.647501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:48.647504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:48.647535 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:48.650013 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:48.650677 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:48.650681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:48.650683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:48.650808 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:48.651131 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:48.651134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:48.651136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:48.651385 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:48.651493 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:48.652042 (+ 549us) server_negotiation.cc:299] Negotiation successful
0809 04:54:48.652200 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:54:48.653266 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:48.636772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:34939 (local address 127.31.93.3:38069)
0809 04:54:48.637539 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:48.637585 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:48.639306 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:48.639934 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:48.639941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:48.640417 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:48.640943 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:48.640960 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.642364 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.642368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:48.643544 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:48.643561 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.644464 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.644468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:48.644654 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:48.645249 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:54:48.645270 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:48.647316 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:48.650126 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:48.650133 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:48.650142 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:48.650592 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:48.650885 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:54:48.650888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:54:48.650890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:54:48.651052 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:54:48.651470 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:48.651475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:48.651742 (+ 267us) client_negotiation.cc:769] Sending connection context
0809 04:54:48.651956 (+ 214us) client_negotiation.cc:240] Negotiation successful
0809 04:54:48.652285 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":310,"threads_started":1}
I20260809 04:54:48.654476 31065 heartbeater.cc:347] Connected to a master server at 127.31.93.62:34939
I20260809 04:54:48.654750 31065 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:48.655401 31065 heartbeater.cc:511] Master 127.31.93.62:34939 requested a full tablet report, sending...
I20260809 04:54:48.656690 30261 ts_manager.cc:194] Registered new tserver with Master: 1b7efb73f2114b88869101753a51c8c1 (127.31.93.3:33409)
I20260809 04:54:48.657435 30261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.93.3@KRBTEST.COM'} at 127.31.93.3:38069
I20260809 04:54:48.661247 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:54:48.670661 30591 heartbeater.cc:503] Master 127.31.93.62:34939 was elected leader, sending a full tablet report...
I20260809 04:54:48.676473 30594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:48.665107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.31.93.62:34939)
0809 04:54:48.665308 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:48.665314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:48.665669 (+ 355us) server_negotiation.cc:407] Connection header received
0809 04:54:48.665791 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:48.665794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:48.665849 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:48.665920 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:48.667211 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.667894 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.669049 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.669407 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.669490 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:48.671682 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:48.671686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:48.671689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:48.671719 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:48.674170 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:48.674838 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:48.674841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:48.674844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:48.674995 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:48.675294 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:48.675297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:48.675299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:48.675594 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:48.675766 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:48.676033 (+ 267us) server_negotiation.cc:299] Negotiation successful
0809 04:54:48.676248 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:54:48.679899 30261 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:54:48.686470 30261 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:54:48.708127 31089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:48.698101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.31.93.3:33409)
0809 04:54:48.699925 (+ 1824us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:48.699935 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:48.699981 (+ 46us) server_negotiation.cc:407] Connection header received
0809 04:54:48.700086 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:48.700094 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:48.700348 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:48.700547 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:48.701246 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.702835 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.705244 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.706366 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.706493 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:48.706731 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:48.707643 (+ 912us) server_negotiation.cc:299] Negotiation successful
0809 04:54:48.707807 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1643,"thread_start_us":1482,"threads_started":1}
I20260809 04:54:48.708288 31087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:48.697913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:33409 (local address 127.0.0.1:57144)
0809 04:54:48.699214 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:48.699271 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:48.699440 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:48.700619 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:48.700627 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:48.700656 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:48.701034 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:48.701047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.703009 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.703014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:48.705053 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:48.705067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.706600 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.706608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:48.707343 (+ 735us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:48.707414 (+ 71us) client_negotiation.cc:769] Sending connection context
0809 04:54:48.707600 (+ 186us) client_negotiation.cc:240] Negotiation successful
0809 04:54:48.707802 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":26,"thread_start_us":884,"threads_started":1}
I20260809 04:54:48.708588 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:48.698344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:44077 (local address 127.0.0.1:38974)
0809 04:54:48.700399 (+ 2055us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:48.700413 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:48.700520 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:48.701529 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:48.701534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:48.701558 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:48.701935 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:48.701947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.703794 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.703805 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:48.705935 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:48.705953 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.707757 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.707763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:48.707917 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:48.707959 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 04:54:48.708043 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 04:54:48.708163 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1965,"thread_start_us":1842,"threads_started":1}
I20260809 04:54:48.709522 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:48.699003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38974 (local address 127.31.93.1:44077)
0809 04:54:48.700239 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:48.700248 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:48.700500 (+ 252us) server_negotiation.cc:407] Connection header received
0809 04:54:48.700880 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:48.700888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:48.701133 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:48.701304 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:48.702136 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.703666 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.706109 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.707509 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.708818 (+ 1309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:48.709092 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:48.709158 (+ 66us) server_negotiation.cc:299] Negotiation successful
0809 04:54:48.709279 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1062,"thread_start_us":920,"threads_started":1}
I20260809 04:54:48.710606 31092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:48.700591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:35049 (local address 127.0.0.1:41600)
0809 04:54:48.702847 (+ 2256us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:48.702863 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:48.702969 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:48.703524 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:48.703528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:48.703553 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:48.703973 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:48.703989 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.705892 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.705898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:48.707618 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:48.707632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.709690 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.709697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:48.709873 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:48.709910 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 04:54:48.710012 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 04:54:48.710138 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2136,"mutex_wait_us":46,"thread_start_us":2022,"threads_started":1}
I20260809 04:54:48.710862 31094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:48.701088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.31.93.2:35049)
0809 04:54:48.702470 (+ 1382us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:48.702477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:48.702957 (+ 480us) server_negotiation.cc:407] Connection header received
0809 04:54:48.703105 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:48.703113 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:48.703324 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:48.703521 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:48.704173 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.705720 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.707807 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:48.709475 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:48.709606 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:48.709785 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:48.710426 (+ 641us) server_negotiation.cc:299] Negotiation successful
0809 04:54:48.710588 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1233,"thread_start_us":1124,"threads_started":1}
I20260809 04:54:48.712911 30471 tablet_service.cc:1467] Processing CreateTablet for tablet eccd81c807b1416492faf1eea14a11ec (DEFAULT_TABLE table=test-table [id=b879d9f774654a2eb6264404fc53ecb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:48.712935 30952 tablet_service.cc:1467] Processing CreateTablet for tablet eccd81c807b1416492faf1eea14a11ec (DEFAULT_TABLE table=test-table [id=b879d9f774654a2eb6264404fc53ecb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:48.714219 30471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eccd81c807b1416492faf1eea14a11ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:48.714367 30952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eccd81c807b1416492faf1eea14a11ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:48.714179 30697 tablet_service.cc:1467] Processing CreateTablet for tablet eccd81c807b1416492faf1eea14a11ec (DEFAULT_TABLE table=test-table [id=b879d9f774654a2eb6264404fc53ecb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:48.715435 30697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eccd81c807b1416492faf1eea14a11ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:48.725747 31095 tablet_bootstrap.cc:492] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376: Bootstrap starting.
I20260809 04:54:48.729564 31098 tablet_bootstrap.cc:492] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1: Bootstrap starting.
I20260809 04:54:48.729600 31095 tablet_bootstrap.cc:654] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:48.730933 31096 tablet_bootstrap.cc:492] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0: Bootstrap starting.
I20260809 04:54:48.730986 31095 log.cc:824] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:48.733459 31098 tablet_bootstrap.cc:654] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:48.734177 31096 tablet_bootstrap.cc:654] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:48.734957 31098 log.cc:824] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:48.735500 31096 log.cc:824] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:48.737715 31095 tablet_bootstrap.cc:492] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376: No bootstrap required, opened a new log
I20260809 04:54:48.738044 31095 ts_tablet_manager.cc:1397] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260809 04:54:48.738334 31096 tablet_bootstrap.cc:492] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0: No bootstrap required, opened a new log
I20260809 04:54:48.738577 31096 ts_tablet_manager.cc:1397] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260809 04:54:48.745375 31098 tablet_bootstrap.cc:492] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1: No bootstrap required, opened a new log
I20260809 04:54:48.745673 31098 ts_tablet_manager.cc:1397] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260809 04:54:48.745903 31096 raft_consensus.cc:348] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7efb73f2114b88869101753a51c8c1" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 33409 } } peers { permanent_uuid: "c5bf11bd406a40e1913a51bcdd6fb376" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 44077 } } peers { permanent_uuid: "72626b15e1de4418a5a80f522e3512d0" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35049 } }
I20260809 04:54:48.746250 31096 raft_consensus.cc:374] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:48.746449 31096 raft_consensus.cc:729] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72626b15e1de4418a5a80f522e3512d0, State: Initialized, Role: FOLLOWER
I20260809 04:54:48.747581 31096 consensus_queue.cc:260] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7efb73f2114b88869101753a51c8c1" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 33409 } } peers { permanent_uuid: "c5bf11bd406a40e1913a51bcdd6fb376" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 44077 } } peers { permanent_uuid: "72626b15e1de4418a5a80f522e3512d0" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35049 } }
I20260809 04:54:48.748832 30825 heartbeater.cc:503] Master 127.31.93.62:34939 was elected leader, sending a full tablet report...
I20260809 04:54:48.749279 31096 ts_tablet_manager.cc:1428] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260809 04:54:48.749779 31095 raft_consensus.cc:348] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7efb73f2114b88869101753a51c8c1" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 33409 } } peers { permanent_uuid: "c5bf11bd406a40e1913a51bcdd6fb376" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 44077 } } peers { permanent_uuid: "72626b15e1de4418a5a80f522e3512d0" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35049 } }
I20260809 04:54:48.750183 31095 raft_consensus.cc:374] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:48.750319 31095 raft_consensus.cc:729] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5bf11bd406a40e1913a51bcdd6fb376, State: Initialized, Role: FOLLOWER
I20260809 04:54:48.750864 31095 consensus_queue.cc:260] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7efb73f2114b88869101753a51c8c1" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 33409 } } peers { permanent_uuid: "c5bf11bd406a40e1913a51bcdd6fb376" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 44077 } } peers { permanent_uuid: "72626b15e1de4418a5a80f522e3512d0" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35049 } }
I20260809 04:54:48.752444 31095 ts_tablet_manager.cc:1428] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260809 04:54:48.756387 31098 raft_consensus.cc:348] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7efb73f2114b88869101753a51c8c1" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 33409 } } peers { permanent_uuid: "c5bf11bd406a40e1913a51bcdd6fb376" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 44077 } } peers { permanent_uuid: "72626b15e1de4418a5a80f522e3512d0" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35049 } }
I20260809 04:54:48.756738 31098 raft_consensus.cc:374] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:48.756819 31098 raft_consensus.cc:729] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b7efb73f2114b88869101753a51c8c1, State: Initialized, Role: FOLLOWER
I20260809 04:54:48.757313 31098 consensus_queue.cc:260] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7efb73f2114b88869101753a51c8c1" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 33409 } } peers { permanent_uuid: "c5bf11bd406a40e1913a51bcdd6fb376" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 44077 } } peers { permanent_uuid: "72626b15e1de4418a5a80f522e3512d0" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35049 } }
I20260809 04:54:48.758229 31065 heartbeater.cc:503] Master 127.31.93.62:34939 was elected leader, sending a full tablet report...
I20260809 04:54:48.758853 31098 ts_tablet_manager.cc:1428] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
W20260809 04:54:48.885254 31066 tablet.cc:2369] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:54:48.888094 30826 tablet.cc:2369] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 04:54:48.893131 30593 tablet.cc:2369] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:54:49.042860 31103 raft_consensus.cc:482] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:54:49.043176 31103 raft_consensus.cc:504] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7efb73f2114b88869101753a51c8c1" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 33409 } } peers { permanent_uuid: "c5bf11bd406a40e1913a51bcdd6fb376" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 44077 } } peers { permanent_uuid: "72626b15e1de4418a5a80f522e3512d0" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35049 } }
I20260809 04:54:49.044936 31103 leader_election.cc:288] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b7efb73f2114b88869101753a51c8c1 (127.31.93.3:33409), 72626b15e1de4418a5a80f522e3512d0 (127.31.93.2:35049)
I20260809 04:54:49.051599 31089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:49.045442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:44109 (local address 127.31.93.3:33409)
0809 04:54:49.045648 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:49.045654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:49.046516 (+ 862us) server_negotiation.cc:407] Connection header received
0809 04:54:49.046710 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:49.046714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:49.046775 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:49.046852 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:49.047421 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:49.048156 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:49.049928 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:49.050737 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:49.050849 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:49.050949 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:49.051228 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 04:54:49.051349 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 04:54:49.051672 31094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:49.045510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:46719 (local address 127.31.93.2:35049)
0809 04:54:49.045724 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:49.045730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:49.046569 (+ 839us) server_negotiation.cc:407] Connection header received
0809 04:54:49.046733 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:49.046738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:49.046798 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:49.046871 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:49.047519 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:49.048223 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:49.049954 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:49.050787 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:49.050903 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:49.050997 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:49.051288 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 04:54:49.051409 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 04:54:49.051766 31107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:49.045418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:33409 (local address 127.31.93.1:44109)
0809 04:54:49.046399 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:49.046419 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:49.046570 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:49.046876 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:49.046880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:49.046914 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:49.047275 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:49.047285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:49.048277 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:49.048282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:49.049778 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:49.049790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:49.050936 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:49.050947 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:49.051084 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:49.051108 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:54:49.051189 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 04:54:49.051309 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":1,"thread_start_us":505,"threads_started":1}
I20260809 04:54:49.051792 31111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:49.045442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:35049 (local address 127.31.93.1:46719)
0809 04:54:49.046427 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:49.046446 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:49.046598 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:49.046903 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:49.046907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:49.046931 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:49.047356 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:49.047365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:49.048360 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:49.048364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:49.049801 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:49.049813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:49.051025 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:49.051032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:49.051146 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:49.051167 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 04:54:49.051245 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 04:54:49.051361 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":6,"spinlock_wait_cycles":55552,"thread_start_us":633,"threads_started":1}
I20260809 04:54:49.052850 30991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eccd81c807b1416492faf1eea14a11ec" candidate_uuid: "c5bf11bd406a40e1913a51bcdd6fb376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b7efb73f2114b88869101753a51c8c1" is_pre_election: true
I20260809 04:54:49.052953 30739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eccd81c807b1416492faf1eea14a11ec" candidate_uuid: "c5bf11bd406a40e1913a51bcdd6fb376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72626b15e1de4418a5a80f522e3512d0" is_pre_election: true
I20260809 04:54:49.053436 30739 raft_consensus.cc:2454] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5bf11bd406a40e1913a51bcdd6fb376 in term 0.
I20260809 04:54:49.053436 30991 raft_consensus.cc:2454] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5bf11bd406a40e1913a51bcdd6fb376 in term 0.
I20260809 04:54:49.054171 30388 leader_election.cc:302] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72626b15e1de4418a5a80f522e3512d0, c5bf11bd406a40e1913a51bcdd6fb376; no voters:
I20260809 04:54:49.054617 31103 raft_consensus.cc:2781] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:54:49.054760 31103 raft_consensus.cc:482] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:54:49.054862 31103 raft_consensus.cc:3037] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:49.057096 31103 raft_consensus.cc:504] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7efb73f2114b88869101753a51c8c1" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 33409 } } peers { permanent_uuid: "c5bf11bd406a40e1913a51bcdd6fb376" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 44077 } } peers { permanent_uuid: "72626b15e1de4418a5a80f522e3512d0" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35049 } }
I20260809 04:54:49.057718 31103 leader_election.cc:288] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [CANDIDATE]: Term 1 election: Requested vote from peers 1b7efb73f2114b88869101753a51c8c1 (127.31.93.3:33409), 72626b15e1de4418a5a80f522e3512d0 (127.31.93.2:35049)
I20260809 04:54:49.058053 30991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eccd81c807b1416492faf1eea14a11ec" candidate_uuid: "c5bf11bd406a40e1913a51bcdd6fb376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b7efb73f2114b88869101753a51c8c1"
I20260809 04:54:49.058188 30739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eccd81c807b1416492faf1eea14a11ec" candidate_uuid: "c5bf11bd406a40e1913a51bcdd6fb376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72626b15e1de4418a5a80f522e3512d0"
I20260809 04:54:49.058276 30991 raft_consensus.cc:3037] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:49.058413 30739 raft_consensus.cc:3037] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:49.061389 30739 raft_consensus.cc:2454] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5bf11bd406a40e1913a51bcdd6fb376 in term 1.
I20260809 04:54:49.061415 30991 raft_consensus.cc:2454] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5bf11bd406a40e1913a51bcdd6fb376 in term 1.
I20260809 04:54:49.061791 30388 leader_election.cc:302] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72626b15e1de4418a5a80f522e3512d0, c5bf11bd406a40e1913a51bcdd6fb376; no voters:
I20260809 04:54:49.062065 31103 raft_consensus.cc:2781] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:54:49.062525 31103 raft_consensus.cc:686] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [term 1 LEADER]: Becoming Leader. State: Replica: c5bf11bd406a40e1913a51bcdd6fb376, State: Running, Role: LEADER
I20260809 04:54:49.063006 31103 consensus_queue.cc:237] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7efb73f2114b88869101753a51c8c1" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 33409 } } peers { permanent_uuid: "c5bf11bd406a40e1913a51bcdd6fb376" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 44077 } } peers { permanent_uuid: "72626b15e1de4418a5a80f522e3512d0" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35049 } }
I20260809 04:54:49.067600 30261 catalog_manager.cc:5495] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5bf11bd406a40e1913a51bcdd6fb376 (127.31.93.1). New cstate: current_term: 1 leader_uuid: "c5bf11bd406a40e1913a51bcdd6fb376" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7efb73f2114b88869101753a51c8c1" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5bf11bd406a40e1913a51bcdd6fb376" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 44077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72626b15e1de4418a5a80f522e3512d0" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35049 } health_report { overall_health: UNKNOWN } } }
I20260809 04:54:49.160782 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:49.155501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38986 (local address 127.31.93.1:44077)
0809 04:54:49.155724 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:49.155730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:49.155749 (+ 19us) server_negotiation.cc:407] Connection header received
0809 04:54:49.155914 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:49.155918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:49.155996 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:49.156137 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:54:49.156512 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:49.157240 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:49.158667 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:49.159083 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:49.159196 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:49.159669 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:54:49.160188 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:54:49.160317 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:49.160447 (+ 130us) server_negotiation.cc:299] Negotiation successful
0809 04:54:49.160530 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 04:54:49.169003 30991 raft_consensus.cc:1261] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1 [term 1 FOLLOWER]: Refusing update from remote peer c5bf11bd406a40e1913a51bcdd6fb376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:54:49.169000 30739 raft_consensus.cc:1261] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0 [term 1 FOLLOWER]: Refusing update from remote peer c5bf11bd406a40e1913a51bcdd6fb376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:54:49.169989 31103 consensus_queue.cc:1035] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72626b15e1de4418a5a80f522e3512d0" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:54:49.170272 31112 consensus_queue.cc:1035] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b7efb73f2114b88869101753a51c8c1" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:54:49.196492 30259 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:54:49.196743 30259 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:54:49.200104 30259 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b7cda3236cb48529b69b28df6676da1: Sending DeleteTablet for 3 replicas of tablet eccd81c807b1416492faf1eea14a11ec
I20260809 04:54:49.201125 30952 tablet_service.cc:1514] Processing DeleteTablet for tablet eccd81c807b1416492faf1eea14a11ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:49 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:57144
I20260809 04:54:49.201197 30471 tablet_service.cc:1514] Processing DeleteTablet for tablet eccd81c807b1416492faf1eea14a11ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:49 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:38974
I20260809 04:54:49.201792 30697 tablet_service.cc:1514] Processing DeleteTablet for tablet eccd81c807b1416492faf1eea14a11ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:49 UTC) from {username='kudu', principal='kudu/127.31.93.62@KRBTEST.COM'} at 127.0.0.1:41600
I20260809 04:54:49.201939 31135 tablet_replica.cc:330] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1: stopping tablet replica
I20260809 04:54:49.202343 31135 raft_consensus.cc:2229] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:54:49.202754 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 30365
I20260809 04:54:49.202809 31135 raft_consensus.cc:2258] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:49.203722 31137 tablet_replica.cc:330] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0: stopping tablet replica
I20260809 04:54:49.203999 31137 raft_consensus.cc:2229] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:54:49.204428 31137 raft_consensus.cc:2258] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:49.206034 31135 ts_tablet_manager.cc:1905] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:54:49.209575 31136 tablet_replica.cc:330] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376: stopping tablet replica
I20260809 04:54:49.209954 31136 raft_consensus.cc:2229] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:54:49.210664 31136 raft_consensus.cc:2258] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:49.212666 31137 ts_tablet_manager.cc:1905] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:54:49.212808 31135 ts_tablet_manager.cc:1918] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:54:49.213060 31135 log.cc:1196] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/eccd81c807b1416492faf1eea14a11ec
I20260809 04:54:49.213224 31136 ts_tablet_manager.cc:1905] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:54:49.213649 31135 ts_tablet_manager.cc:1939] T eccd81c807b1416492faf1eea14a11ec P 1b7efb73f2114b88869101753a51c8c1: Deleting consensus metadata
I20260809 04:54:49.215476 30236 catalog_manager.cc:4841] TS 1b7efb73f2114b88869101753a51c8c1 (127.31.93.3:33409): tablet eccd81c807b1416492faf1eea14a11ec (table test-table [id=b879d9f774654a2eb6264404fc53ecb8]) successfully deleted
Aug 09 04:54:49 dist-test-slave-4tdj krb5kdc[30181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.1@KRBTEST.COM
I20260809 04:54:49.219668 31137 ts_tablet_manager.cc:1918] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:54:49.220158 31137 log.cc:1196] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/eccd81c807b1416492faf1eea14a11ec
I20260809 04:54:49.220752 31137 ts_tablet_manager.cc:1939] T eccd81c807b1416492faf1eea14a11ec P 72626b15e1de4418a5a80f522e3512d0: Deleting consensus metadata
I20260809 04:54:49.222484 30237 catalog_manager.cc:4841] TS 72626b15e1de4418a5a80f522e3512d0 (127.31.93.2:35049): tablet eccd81c807b1416492faf1eea14a11ec (table test-table [id=b879d9f774654a2eb6264404fc53ecb8]) successfully deleted
I20260809 04:54:49.222978 31136 ts_tablet_manager.cc:1918] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:54:49.223150 31136 log.cc:1196] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/eccd81c807b1416492faf1eea14a11ec
I20260809 04:54:49.223639 31136 ts_tablet_manager.cc:1939] T eccd81c807b1416492faf1eea14a11ec P c5bf11bd406a40e1913a51bcdd6fb376: Deleting consensus metadata
I20260809 04:54:49.225135 30237 catalog_manager.cc:4841] TS c5bf11bd406a40e1913a51bcdd6fb376 (127.31.93.1:44077): tablet eccd81c807b1416492faf1eea14a11ec (table test-table [id=b879d9f774654a2eb6264404fc53ecb8]) successfully deleted
I20260809 04:54:49.225239 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:49.204213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39000 (local address 127.31.93.1:44077)
0809 04:54:49.204384 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:49.204390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:49.204455 (+ 65us) server_negotiation.cc:407] Connection header received
0809 04:54:49.204730 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:49.204735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:49.204811 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:49.204889 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:49.206900 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:49.208099 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:49.209868 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:49.210568 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:49.210667 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:49.217766 (+ 7099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:49.217780 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:49.217796 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:49.217849 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:49.221780 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:49.222736 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:49.222745 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:49.222748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:49.222930 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:49.223437 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:49.223442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:49.223446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:49.223912 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:49.224228 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:49.224608 (+ 380us) server_negotiation.cc:299] Negotiation successful
0809 04:54:49.224859 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 04:54:49.225919 30585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:49.320394 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 30365
I20260809 04:54:49.343837 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 30601
Aug 09 04:54:49 dist-test-slave-4tdj krb5kdc[30181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.2@KRBTEST.COM
I20260809 04:54:49.357954 31094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:49.344575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.31.93.2:35049)
0809 04:54:49.344745 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:49.344751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:49.344771 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:54:49.344960 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:49.344964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:49.345028 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:49.345113 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:49.346390 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:49.347117 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:49.348384 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:49.348775 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:49.348882 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:49.352853 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:49.352866 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:49.352881 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:49.352918 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:49.355399 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:49.356013 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:49.356020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:49.356022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:49.356149 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:49.356487 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:49.356490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:49.356492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:49.356862 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:49.357145 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:49.357422 (+ 277us) server_negotiation.cc:299] Negotiation successful
0809 04:54:49.357635 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:54:49.358666 30811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:49.443783 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 30601
I20260809 04:54:49.466624 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 30833
Aug 09 04:54:49 dist-test-slave-4tdj krb5kdc[30181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.3@KRBTEST.COM
I20260809 04:54:49.480854 31089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:49.467427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.31.93.3:33409)
0809 04:54:49.467617 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:49.467622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:49.467642 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:54:49.467763 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:49.467767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:49.467829 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:49.467904 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:49.469110 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:49.469804 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:49.471082 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:49.471478 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:49.471597 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:49.475906 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:49.475938 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:49.475955 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:49.475994 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:49.478609 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:49.479193 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:49.479200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:49.479202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:49.479334 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:49.479620 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:49.479623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:49.479625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:49.479960 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:49.480168 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:49.480359 (+ 191us) server_negotiation.cc:299] Negotiation successful
0809 04:54:49.480512 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:54:49.481472 31057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:49.566597 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 30833
I20260809 04:54:49.589721 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 30201
I20260809 04:54:49.590795 30310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:49.696902 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 30201
2026-08-09T04:54:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4556 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-09T04:54:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:54:49Z Disabled control of system clock
I20260809 04:54:49.783908 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:33973
--webserver_interface=127.31.93.62
--webserver_port=0
--builtin_ntp_servers=127.31.93.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:33973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 04:54:50.114786 31157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:50.115062 31157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:50.115195 31157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:50.122351 31157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:54:50.122449 31157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:50.122512 31157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:50.122589 31157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:54:50.122659 31157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:54:50.131634 31157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:33973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:33973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:50.132880 31157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:50.134703 31157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:50.145553 31164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:50.145712 31167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:50.145748 31165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:50.146854 31157 server_base.cc:1034] running on GCE node
I20260809 04:54:50.147595 31157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:50.148962 31157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:50.150161 31157 hybrid_clock.cc:648] HybridClock initialized: now 1786251290150115 us; error 38 us; skew 500 ppm
I20260809 04:54:50.150512 31157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:54:50.152583 31157 webserver.cc:460] Webserver started at http://127.31.93.62:35707/ using document root <none> and password file <none>
I20260809 04:54:50.153264 31157 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:50.153352 31157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:50.153607 31157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:50.156394 31157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "3cf702f1ef56425fbc358c5471eb5e8e"
format_stamp: "Formatted at 2026-08-09 04:54:50 on dist-test-slave-4tdj"
server_key: "c53464ed9a51e366893d8a2b8d87f3e4"
server_key_iv: "95a6d6421a506ef904d26b72d5a0127d"
server_key_version: "encryptionkey@0"
I20260809 04:54:50.157133 31157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "3cf702f1ef56425fbc358c5471eb5e8e"
format_stamp: "Formatted at 2026-08-09 04:54:50 on dist-test-slave-4tdj"
server_key: "c53464ed9a51e366893d8a2b8d87f3e4"
server_key_iv: "95a6d6421a506ef904d26b72d5a0127d"
server_key_version: "encryptionkey@0"
I20260809 04:54:50.162107 31157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 04:54:50.165437 31177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:50.166754 31157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:54:50.166900 31157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "3cf702f1ef56425fbc358c5471eb5e8e"
format_stamp: "Formatted at 2026-08-09 04:54:50 on dist-test-slave-4tdj"
server_key: "c53464ed9a51e366893d8a2b8d87f3e4"
server_key_iv: "95a6d6421a506ef904d26b72d5a0127d"
server_key_version: "encryptionkey@0"
I20260809 04:54:50.167066 31157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:50.180917 31157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:50.181614 31157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:50.181854 31157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:50.198860 31157 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:33973
I20260809 04:54:50.198895 31261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:33973 every 8 connection(s)
I20260809 04:54:50.200130 31157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:54:50.203759 31262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:50.208946 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 31157
I20260809 04:54:50.209111 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:54:50.209537 32116 external_mini_cluster.cc:1428] Setting key ef1e4ec7b07bc94ca317a001a7add9ce
I20260809 04:54:50.215301 31262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e: Bootstrap starting.
I20260809 04:54:50.219179 31262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:50.220340 31262 log.cc:824] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:50.223759 31262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e: No bootstrap required, opened a new log
I20260809 04:54:50.224699 31268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:50.211375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.31.93.62:33973)
0809 04:54:50.212505 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:50.212523 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:50.212574 (+ 51us) server_negotiation.cc:407] Connection header received
0809 04:54:50.214928 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:50.214956 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:50.215521 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:50.215943 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:50.216472 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:50.218418 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:50.219716 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:50.220730 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:50.220927 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:50.222009 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:50.222020 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:50.222039 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:50.222724 (+ 685us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:50.222823 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:50.222851 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:50.222973 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:50.223100 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 04:54:50.223413 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":345,"threads_started":1}
I20260809 04:54:50.231806 31262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf702f1ef56425fbc358c5471eb5e8e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33973 } }
I20260809 04:54:50.232156 31262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:50.232230 31262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf702f1ef56425fbc358c5471eb5e8e, State: Initialized, Role: FOLLOWER
I20260809 04:54:50.232707 31262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf702f1ef56425fbc358c5471eb5e8e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33973 } }
I20260809 04:54:50.232928 31262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:54:50.233067 31262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:54:50.233270 31262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:50.235546 31262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf702f1ef56425fbc358c5471eb5e8e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33973 } }
I20260809 04:54:50.236043 31262 leader_election.cc:302] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cf702f1ef56425fbc358c5471eb5e8e; no voters:
I20260809 04:54:50.236581 31262 leader_election.cc:288] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:54:50.236716 31270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:54:50.237810 31270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [term 1 LEADER]: Becoming Leader. State: Replica: 3cf702f1ef56425fbc358c5471eb5e8e, State: Running, Role: LEADER
I20260809 04:54:50.238415 31270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf702f1ef56425fbc358c5471eb5e8e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33973 } }
I20260809 04:54:50.238710 31262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:54:50.241223 31272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cf702f1ef56425fbc358c5471eb5e8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf702f1ef56425fbc358c5471eb5e8e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33973 } } }
I20260809 04:54:50.241606 31272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:50.242014 31277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cf702f1ef56425fbc358c5471eb5e8e. Latest consensus state: current_term: 1 leader_uuid: "3cf702f1ef56425fbc358c5471eb5e8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf702f1ef56425fbc358c5471eb5e8e" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 33973 } } }
I20260809 04:54:50.242354 31277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:50.243746 31284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:54:50.249073 31284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:54:50.257967 31284 catalog_manager.cc:1347] Generated new cluster ID: e15818d496124b99b0341a1451497065
I20260809 04:54:50.258078 31284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:54:50.276193 31284 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:54:50.277993 31284 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:54:50.285816 31284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e: Generated new TSK 0
I20260809 04:54:50.286651 31284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:54:50.300678 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.1:0
--local_ip_for_outbound_sockets=127.31.93.1
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33973
--builtin_ntp_servers=127.31.93.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 04:54:50.628597 31299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:50.628883 31299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:50.629025 31299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:50.636034 31299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:50.636152 31299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:50.636301 31299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.1
I20260809 04:54:50.645362 31299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.93.1
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.1
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:50.646677 31299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:50.648468 31299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:50.660925 31309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:50.660859 31310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:50.660899 31314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:50.661523 31299 server_base.cc:1034] running on GCE node
I20260809 04:54:50.662096 31299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:50.663138 31299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:50.664342 31299 hybrid_clock.cc:648] HybridClock initialized: now 1786251290664290 us; error 39 us; skew 500 ppm
I20260809 04:54:50.664662 31299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:54:50.666811 31299 webserver.cc:460] Webserver started at http://127.31.93.1:44857/ using document root <none> and password file <none>
I20260809 04:54:50.667440 31299 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:50.667550 31299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:50.667842 31299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:50.670674 31299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/data/instance:
uuid: "939f830253834cba8513a4ebc7360473"
format_stamp: "Formatted at 2026-08-09 04:54:50 on dist-test-slave-4tdj"
server_key: "2959ce0552fcd77e1c5704a8c991cfd5"
server_key_iv: "0f539d71f6d807811cbd2d46f3cde192"
server_key_version: "encryptionkey@0"
I20260809 04:54:50.671448 31299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance:
uuid: "939f830253834cba8513a4ebc7360473"
format_stamp: "Formatted at 2026-08-09 04:54:50 on dist-test-slave-4tdj"
server_key: "2959ce0552fcd77e1c5704a8c991cfd5"
server_key_iv: "0f539d71f6d807811cbd2d46f3cde192"
server_key_version: "encryptionkey@0"
I20260809 04:54:50.676246 31299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:54:50.679553 31327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:50.680739 31299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 04:54:50.680879 31299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/wal
uuid: "939f830253834cba8513a4ebc7360473"
format_stamp: "Formatted at 2026-08-09 04:54:50 on dist-test-slave-4tdj"
server_key: "2959ce0552fcd77e1c5704a8c991cfd5"
server_key_iv: "0f539d71f6d807811cbd2d46f3cde192"
server_key_version: "encryptionkey@0"
I20260809 04:54:50.681061 31299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:50.700065 31299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:50.700830 31299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:50.701092 31299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:50.702003 31299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:50.703434 31299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:50.703529 31299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:50.703636 31299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:50.703706 31299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:50.732952 31299 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.1:39989
I20260809 04:54:50.733026 31518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.1:39989 every 8 connection(s)
I20260809 04:54:50.734272 31299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/data/info.pb
I20260809 04:54:50.740833 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 31299
I20260809 04:54:50.741004 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/wal/instance
I20260809 04:54:50.741490 32116 external_mini_cluster.cc:1428] Setting key 0373e42f78d6fd54367d2e82e3bbe5ff
I20260809 04:54:50.744215 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.2:0
--local_ip_for_outbound_sockets=127.31.93.2
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33973
--builtin_ntp_servers=127.31.93.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 04:54:50.748472 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:50.736776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.1:51767 (local address 127.31.93.62:33973)
0809 04:54:50.737217 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:50.737221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:50.738423 (+ 1202us) server_negotiation.cc:407] Connection header received
0809 04:54:50.742068 (+ 3645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:50.742072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:50.742150 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:50.742308 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:50.743518 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:50.744282 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:50.745840 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:50.746402 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:50.746535 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:50.747744 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:50.747750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:50.747754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:50.747775 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:50.747820 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:50.747834 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:50.747913 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:50.748083 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 04:54:50.748171 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":225,"threads_started":1}
I20260809 04:54:50.749300 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:50.737241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:33973 (local address 127.31.93.1:51767)
0809 04:54:50.738252 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:50.738321 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:50.740285 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:50.742484 (+ 2199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:50.742492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:50.742598 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:50.743359 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:50.743379 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:50.744479 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:50.744483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:50.745698 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:50.745711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:50.746592 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:50.746597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:50.746813 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:50.747430 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:54:50.747432 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:50.747474 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:54:50.747548 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:50.747921 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:50.747927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:50.747932 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 04:54:50.748054 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 04:54:50.748322 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":263,"threads_started":1}
I20260809 04:54:50.750805 31520 heartbeater.cc:347] Connected to a master server at 127.31.93.62:33973
I20260809 04:54:50.751075 31520 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:50.751755 31520 heartbeater.cc:511] Master 127.31.93.62:33973 requested a full tablet report, sending...
I20260809 04:54:50.753895 31205 ts_manager.cc:194] Registered new tserver with Master: 939f830253834cba8513a4ebc7360473 (127.31.93.1:39989)
I20260809 04:54:50.755759 31205 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.93.1:51767
W20260809 04:54:51.069806 31527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:51.070072 31527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:51.070199 31527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:51.077239 31527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:51.077327 31527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:51.077459 31527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.2
I20260809 04:54:51.086050 31527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.93.2
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.2
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:51.087215 31527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:51.088943 31527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:51.101022 31536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:51.101107 31537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:51.101065 31540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:51.101970 31527 server_base.cc:1034] running on GCE node
I20260809 04:54:51.102557 31527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:51.103363 31527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:51.104573 31527 hybrid_clock.cc:648] HybridClock initialized: now 1786251291104521 us; error 41 us; skew 500 ppm
I20260809 04:54:51.104862 31527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:54:51.106972 31527 webserver.cc:460] Webserver started at http://127.31.93.2:33493/ using document root <none> and password file <none>
I20260809 04:54:51.107594 31527 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:51.107690 31527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:51.107973 31527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:51.110697 31527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/data/instance:
uuid: "32df36ed782f47d6ab892190ac369a8f"
format_stamp: "Formatted at 2026-08-09 04:54:51 on dist-test-slave-4tdj"
server_key: "20b13109d37c5745d14df944bc8b9b36"
server_key_iv: "0ff292703b64c437ea2d9baa026a368e"
server_key_version: "encryptionkey@0"
I20260809 04:54:51.111443 31527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance:
uuid: "32df36ed782f47d6ab892190ac369a8f"
format_stamp: "Formatted at 2026-08-09 04:54:51 on dist-test-slave-4tdj"
server_key: "20b13109d37c5745d14df944bc8b9b36"
server_key_iv: "0ff292703b64c437ea2d9baa026a368e"
server_key_version: "encryptionkey@0"
I20260809 04:54:51.116158 31527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:54:51.119313 31554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:51.120520 31527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 04:54:51.120662 31527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/wal
uuid: "32df36ed782f47d6ab892190ac369a8f"
format_stamp: "Formatted at 2026-08-09 04:54:51 on dist-test-slave-4tdj"
server_key: "20b13109d37c5745d14df944bc8b9b36"
server_key_iv: "0ff292703b64c437ea2d9baa026a368e"
server_key_version: "encryptionkey@0"
I20260809 04:54:51.120850 31527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:51.131304 31527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:51.131964 31527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:51.132205 31527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:51.133070 31527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:51.134467 31527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:51.134562 31527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:51.134675 31527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:51.134747 31527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:51.163820 31527 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.2:45883
I20260809 04:54:51.163863 31757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.2:45883 every 8 connection(s)
I20260809 04:54:51.165027 31527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/data/info.pb
I20260809 04:54:51.172183 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 31527
I20260809 04:54:51.172365 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/wal/instance
I20260809 04:54:51.172883 32116 external_mini_cluster.cc:1428] Setting key 0a9b1b23f9567d6ffb67d36e96a1b11c
I20260809 04:54:51.175758 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.93.3:0
--local_ip_for_outbound_sockets=127.31.93.3
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33973
--builtin_ntp_servers=127.31.93.20:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 04:54:51.177014 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.167445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.2:46565 (local address 127.31.93.62:33973)
0809 04:54:51.167660 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:51.167665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:51.168904 (+ 1239us) server_negotiation.cc:407] Connection header received
0809 04:54:51.170630 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:51.170635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:51.170726 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:51.170805 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:51.171817 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.172844 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.174339 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.174862 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.174955 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.176374 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:51.176378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:51.176381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:51.176398 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:51.176427 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:51.176438 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:51.176499 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:51.176713 (+ 214us) server_negotiation.cc:299] Negotiation successful
0809 04:54:51.176784 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 04:54:51.178218 31762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.167895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:33973 (local address 127.31.93.2:46565)
0809 04:54:51.168762 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:51.168802 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:51.170408 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:51.170987 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:51.170994 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:51.171090 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:51.171678 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:51.171697 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.173001 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.173006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:51.174205 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:51.174217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.175078 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.175086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:51.175352 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.176058 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:54:51.176061 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:51.176112 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:54:51.176192 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:51.176516 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:51.176525 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:51.176533 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 04:54:51.176715 (+ 182us) client_negotiation.cc:240] Negotiation successful
0809 04:54:51.177065 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":280,"threads_started":1}
I20260809 04:54:51.179486 31758 heartbeater.cc:347] Connected to a master server at 127.31.93.62:33973
I20260809 04:54:51.179767 31758 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:51.180524 31758 heartbeater.cc:511] Master 127.31.93.62:33973 requested a full tablet report, sending...
I20260809 04:54:51.181893 31205 ts_manager.cc:194] Registered new tserver with Master: 32df36ed782f47d6ab892190ac369a8f (127.31.93.2:45883)
I20260809 04:54:51.182716 31205 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.93.2:46565
W20260809 04:54:51.495429 31764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:51.495677 31764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:51.495766 31764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:51.502980 31764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:51.503062 31764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:51.503163 31764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.93.3
I20260809 04:54:51.511602 31764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.93.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.93.3
--webserver_port=0
--tserver_master_addrs=127.31.93.62:33973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.93.3
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:51.512656 31764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:51.514256 31764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:51.526268 31772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:51.526346 31771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:51.526326 31774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:51.531464 31764 server_base.cc:1034] running on GCE node
I20260809 04:54:51.532011 31764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:51.532828 31764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:51.534010 31764 hybrid_clock.cc:648] HybridClock initialized: now 1786251291533951 us; error 57 us; skew 500 ppm
I20260809 04:54:51.534313 31764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 04:54:51.536317 31764 webserver.cc:460] Webserver started at http://127.31.93.3:46013/ using document root <none> and password file <none>
I20260809 04:54:51.536947 31764 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:51.537045 31764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:51.537321 31764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:51.540022 31764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/data/instance:
uuid: "f66bd4063ca7492f944f50db6d2ec725"
format_stamp: "Formatted at 2026-08-09 04:54:51 on dist-test-slave-4tdj"
server_key: "a0cf5f11d43e3ebd2931f7310eaa93d1"
server_key_iv: "129db26c814f3df3ef8678ff1e158907"
server_key_version: "encryptionkey@0"
I20260809 04:54:51.540768 31764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance:
uuid: "f66bd4063ca7492f944f50db6d2ec725"
format_stamp: "Formatted at 2026-08-09 04:54:51 on dist-test-slave-4tdj"
server_key: "a0cf5f11d43e3ebd2931f7310eaa93d1"
server_key_iv: "129db26c814f3df3ef8678ff1e158907"
server_key_version: "encryptionkey@0"
I20260809 04:54:51.545476 31764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:54:51.548606 31785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:51.549770 31764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 04:54:51.549916 31764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/wal
uuid: "f66bd4063ca7492f944f50db6d2ec725"
format_stamp: "Formatted at 2026-08-09 04:54:51 on dist-test-slave-4tdj"
server_key: "a0cf5f11d43e3ebd2931f7310eaa93d1"
server_key_iv: "129db26c814f3df3ef8678ff1e158907"
server_key_version: "encryptionkey@0"
I20260809 04:54:51.550092 31764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:51.560698 31764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:51.561352 31764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:51.561582 31764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:51.562497 31764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 04:54:51.563777 31764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 04:54:51.563869 31764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:51.563969 31764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 04:54:51.564038 31764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:51.595245 31764 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.3:40855
I20260809 04:54:51.595291 31965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.3:40855 every 8 connection(s)
I20260809 04:54:51.596438 31764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/data/info.pb
I20260809 04:54:51.603973 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 31764
I20260809 04:54:51.604166 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/wal/instance
I20260809 04:54:51.604732 32116 external_mini_cluster.cc:1428] Setting key 8ae5753bfe141497031bdd1b2480b9fb
I20260809 04:54:51.608269 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.598759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:42793 (local address 127.31.93.62:33973)
0809 04:54:51.598942 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:51.598947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:51.600184 (+ 1237us) server_negotiation.cc:407] Connection header received
0809 04:54:51.601790 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:51.601794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:51.601863 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:51.601945 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:51.602922 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.603656 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.605708 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.606214 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.606373 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.607665 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:51.607670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:51.607673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:51.607689 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:51.607720 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:51.607730 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:51.607794 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:51.607963 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 04:54:51.608034 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 04:54:51.609104 31968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.599197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:33973 (local address 127.31.93.3:42793)
0809 04:54:51.600032 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:51.600073 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:51.601571 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:51.602108 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:51.602115 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:51.602199 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:54:51.602789 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:51.602805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.603799 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.603805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:51.605564 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:51.605578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.606454 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.606462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:51.606728 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.607371 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 04:54:51.607373 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:54:51.607419 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 04:54:51.607503 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:54:51.607801 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:54:51.607808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:54:51.607813 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 04:54:51.607957 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 04:54:51.608204 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":262,"threads_started":1}
I20260809 04:54:51.610230 31966 heartbeater.cc:347] Connected to a master server at 127.31.93.62:33973
I20260809 04:54:51.610495 31966 heartbeater.cc:464] Registering TS with master...
I20260809 04:54:51.611135 31966 heartbeater.cc:511] Master 127.31.93.62:33973 requested a full tablet report, sending...
I20260809 04:54:51.612360 31205 ts_manager.cc:194] Registered new tserver with Master: f66bd4063ca7492f944f50db6d2ec725 (127.31.93.3:40855)
I20260809 04:54:51.613093 31205 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.93.3:42793
I20260809 04:54:51.619311 32116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 04:54:51.628216 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.623089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.31.93.62:33973)
0809 04:54:51.623290 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:51.623295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:51.623669 (+ 374us) server_negotiation.cc:407] Connection header received
0809 04:54:51.623826 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:51.623829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:51.623895 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:51.623967 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:51.624400 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.625071 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.626375 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.626749 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.626832 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.627655 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:51.627659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:51.627662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:51.627678 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:51.627706 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:51.627716 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:51.627779 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:51.627878 (+ 99us) server_negotiation.cc:299] Negotiation successful
0809 04:54:51.627951 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:54:51.631377 31205 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 04:54:51.637822 31205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 04:54:51.657958 31988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.648650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.3:40855 (local address 127.0.0.1:52252)
0809 04:54:51.649472 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:51.649513 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:51.649676 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:51.650669 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:51.650677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:51.650708 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:51.651097 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:51.651109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.652914 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.652919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:51.655185 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:51.655200 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.656982 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.656989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:51.657273 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.657312 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 04:54:51.657413 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 04:54:51.657601 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":82,"thread_start_us":1741,"threads_started":1}
I20260809 04:54:51.657972 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.649344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.31.93.3:40855)
0809 04:54:51.650057 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:51.650064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:51.650099 (+ 35us) server_negotiation.cc:407] Connection header received
0809 04:54:51.650200 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:51.650207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:51.650453 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:51.650611 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:51.651299 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.652808 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.655404 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.656783 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.656911 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.657178 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:51.657478 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 04:54:51.657634 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":578,"threads_started":1}
I20260809 04:54:51.659472 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.648761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:39989 (local address 127.0.0.1:43136)
0809 04:54:51.650682 (+ 1921us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:51.650701 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:51.650825 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:51.651627 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:51.651631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:51.651656 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:51.652101 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:51.652114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.653896 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.653901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:51.655791 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:51.655803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.658756 (+ 2953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.658763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:51.658937 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.658968 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 04:54:51.659055 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 04:54:51.659190 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1777,"mutex_wait_us":68,"thread_start_us":376,"threads_started":1}
I20260809 04:54:51.660223 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.651365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:45883 (local address 127.0.0.1:49300)
0809 04:54:51.652100 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:51.652116 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:51.652304 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:51.653138 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:51.653142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:51.653170 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:51.653676 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:51.653688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.655612 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.655617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:51.657603 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:51.657616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.659454 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.659466 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:51.659640 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.659671 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 04:54:51.659759 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 04:54:51.659871 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":75,"thread_start_us":443,"threads_started":1}
I20260809 04:54:51.660243 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.649739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.31.93.1:39989)
0809 04:54:51.650276 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:51.650284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:51.650900 (+ 616us) server_negotiation.cc:407] Connection header received
0809 04:54:51.651001 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:51.651008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:51.651244 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:51.651418 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:51.652289 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.653735 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.655966 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.656841 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.656967 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.657196 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:51.659734 (+ 2538us) server_negotiation.cc:299] Negotiation successful
0809 04:54:51.659909 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":361,"threads_started":1}
I20260809 04:54:51.660300 31994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.651110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49300 (local address 127.31.93.2:45883)
0809 04:54:51.651630 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:51.651638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:51.652207 (+ 569us) server_negotiation.cc:407] Connection header received
0809 04:54:51.652486 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:51.652494 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:51.652705 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:51.652855 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:51.653882 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.655442 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.657774 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.658933 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.659057 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.659318 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:51.659801 (+ 483us) server_negotiation.cc:299] Negotiation successful
0809 04:54:51.659952 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":364,"threads_started":1}
I20260809 04:54:51.662206 31878 tablet_service.cc:1467] Processing CreateTablet for tablet 7047cd0be0384b1f944ffcb1e1362102 (DEFAULT_TABLE table=test-table [id=9c8d9a476e904acfad99d1d58fa6df7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:51.663544 31411 tablet_service.cc:1467] Processing CreateTablet for tablet 7047cd0be0384b1f944ffcb1e1362102 (DEFAULT_TABLE table=test-table [id=9c8d9a476e904acfad99d1d58fa6df7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:51.664171 31878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7047cd0be0384b1f944ffcb1e1362102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:51.663826 31641 tablet_service.cc:1467] Processing CreateTablet for tablet 7047cd0be0384b1f944ffcb1e1362102 (DEFAULT_TABLE table=test-table [id=9c8d9a476e904acfad99d1d58fa6df7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 04:54:51.664873 31411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7047cd0be0384b1f944ffcb1e1362102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:51.664901 31641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7047cd0be0384b1f944ffcb1e1362102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:51.675724 31996 tablet_bootstrap.cc:492] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f: Bootstrap starting.
I20260809 04:54:51.678483 31997 tablet_bootstrap.cc:492] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473: Bootstrap starting.
I20260809 04:54:51.679082 31996 tablet_bootstrap.cc:654] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:51.680397 31996 log.cc:824] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:51.682049 31997 tablet_bootstrap.cc:654] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:51.683343 31997 log.cc:824] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:51.684351 31999 tablet_bootstrap.cc:492] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725: Bootstrap starting.
I20260809 04:54:51.684804 31996 tablet_bootstrap.cc:492] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f: No bootstrap required, opened a new log
I20260809 04:54:51.685067 31996 ts_tablet_manager.cc:1397] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 04:54:51.685891 31997 tablet_bootstrap.cc:492] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473: No bootstrap required, opened a new log
I20260809 04:54:51.686153 31997 ts_tablet_manager.cc:1397] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 04:54:51.687640 31999 tablet_bootstrap.cc:654] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:51.688750 31999 log.cc:824] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:51.693223 31997 raft_consensus.cc:348] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66bd4063ca7492f944f50db6d2ec725" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40855 } } peers { permanent_uuid: "939f830253834cba8513a4ebc7360473" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39989 } } peers { permanent_uuid: "32df36ed782f47d6ab892190ac369a8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 45883 } }
I20260809 04:54:51.693537 31997 raft_consensus.cc:374] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:51.693675 31997 raft_consensus.cc:729] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939f830253834cba8513a4ebc7360473, State: Initialized, Role: FOLLOWER
I20260809 04:54:51.694380 31997 consensus_queue.cc:260] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66bd4063ca7492f944f50db6d2ec725" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40855 } } peers { permanent_uuid: "939f830253834cba8513a4ebc7360473" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39989 } } peers { permanent_uuid: "32df36ed782f47d6ab892190ac369a8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 45883 } }
I20260809 04:54:51.694994 31996 raft_consensus.cc:348] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66bd4063ca7492f944f50db6d2ec725" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40855 } } peers { permanent_uuid: "939f830253834cba8513a4ebc7360473" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39989 } } peers { permanent_uuid: "32df36ed782f47d6ab892190ac369a8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 45883 } }
I20260809 04:54:51.695372 31520 heartbeater.cc:503] Master 127.31.93.62:33973 was elected leader, sending a full tablet report...
I20260809 04:54:51.695485 31996 raft_consensus.cc:374] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:51.695634 31996 raft_consensus.cc:729] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32df36ed782f47d6ab892190ac369a8f, State: Initialized, Role: FOLLOWER
I20260809 04:54:51.696024 31997 ts_tablet_manager.cc:1428] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473: Time spent starting tablet: real 0.010s user 0.002s sys 0.007s
I20260809 04:54:51.696081 31999 tablet_bootstrap.cc:492] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725: No bootstrap required, opened a new log
I20260809 04:54:51.696396 31999 ts_tablet_manager.cc:1397] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260809 04:54:51.696344 31996 consensus_queue.cc:260] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66bd4063ca7492f944f50db6d2ec725" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40855 } } peers { permanent_uuid: "939f830253834cba8513a4ebc7360473" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39989 } } peers { permanent_uuid: "32df36ed782f47d6ab892190ac369a8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 45883 } }
I20260809 04:54:51.697592 31758 heartbeater.cc:503] Master 127.31.93.62:33973 was elected leader, sending a full tablet report...
I20260809 04:54:51.698146 31996 ts_tablet_manager.cc:1428] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260809 04:54:51.704128 31999 raft_consensus.cc:348] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66bd4063ca7492f944f50db6d2ec725" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40855 } } peers { permanent_uuid: "939f830253834cba8513a4ebc7360473" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39989 } } peers { permanent_uuid: "32df36ed782f47d6ab892190ac369a8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 45883 } }
I20260809 04:54:51.704425 31999 raft_consensus.cc:374] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:51.704489 31999 raft_consensus.cc:729] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66bd4063ca7492f944f50db6d2ec725, State: Initialized, Role: FOLLOWER
I20260809 04:54:51.704921 31999 consensus_queue.cc:260] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66bd4063ca7492f944f50db6d2ec725" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40855 } } peers { permanent_uuid: "939f830253834cba8513a4ebc7360473" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39989 } } peers { permanent_uuid: "32df36ed782f47d6ab892190ac369a8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 45883 } }
I20260809 04:54:51.705817 31966 heartbeater.cc:503] Master 127.31.93.62:33973 was elected leader, sending a full tablet report...
I20260809 04:54:51.706372 31999 ts_tablet_manager.cc:1428] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
W20260809 04:54:51.736325 31522 tablet.cc:2369] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:54:51.765403 32019 raft_consensus.cc:482] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 04:54:51.765676 32019 raft_consensus.cc:504] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66bd4063ca7492f944f50db6d2ec725" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40855 } } peers { permanent_uuid: "939f830253834cba8513a4ebc7360473" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39989 } } peers { permanent_uuid: "32df36ed782f47d6ab892190ac369a8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 45883 } }
I20260809 04:54:51.767313 32019 leader_election.cc:288] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 939f830253834cba8513a4ebc7360473 (127.31.93.1:39989), 32df36ed782f47d6ab892190ac369a8f (127.31.93.2:45883)
I20260809 04:54:51.773239 31968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.767705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.1:39989 (local address 127.31.93.3:44909)
0809 04:54:51.768004 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:51.768022 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:51.768158 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:51.768463 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:51.768467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:51.768503 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:51.768697 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:51.768705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.769653 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.769657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:51.771532 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:51.771547 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.772698 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.772703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:51.772816 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.772840 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:54:51.772908 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 04:54:51.773001 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":1,"spinlock_wait_cycles":18816}
I20260809 04:54:51.773342 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.767627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:44909 (local address 127.31.93.1:39989)
0809 04:54:51.767819 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:51.767825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:51.768100 (+ 275us) server_negotiation.cc:407] Connection header received
0809 04:54:51.768281 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:51.768285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:51.768353 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:51.768424 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:51.768850 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.769516 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.771678 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.772566 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.772689 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.772777 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:51.772958 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 04:54:51.773099 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 04:54:51.774119 31994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.767780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.93.3:52189 (local address 127.31.93.2:45883)
0809 04:54:51.768008 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:51.768014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:51.768534 (+ 520us) server_negotiation.cc:407] Connection header received
0809 04:54:51.768714 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:51.768718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:51.768775 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:51.768845 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 04:54:51.769341 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.770030 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.772303 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.773389 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.773505 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.773616 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:51.773858 (+ 242us) server_negotiation.cc:299] Negotiation successful
0809 04:54:51.773946 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 04:54:51.774256 32027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.767773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.2:45883 (local address 127.31.93.3:52189)
0809 04:54:51.768440 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0809 04:54:51.768454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 04:54:51.768588 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:54:51.768872 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:54:51.768877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:54:51.768899 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 04:54:51.769197 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:51.769206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.770409 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.770414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:51.772131 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:54:51.772149 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.773566 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.773572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:54:51.773713 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.773737 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 04:54:51.773827 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 04:54:51.773940 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":24,"thread_start_us":379,"threads_started":1}
I20260809 04:54:51.774266 31439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7047cd0be0384b1f944ffcb1e1362102" candidate_uuid: "f66bd4063ca7492f944f50db6d2ec725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939f830253834cba8513a4ebc7360473" is_pre_election: true
I20260809 04:54:51.774868 31439 raft_consensus.cc:2454] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f66bd4063ca7492f944f50db6d2ec725 in term 0.
I20260809 04:54:51.775178 31681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7047cd0be0384b1f944ffcb1e1362102" candidate_uuid: "f66bd4063ca7492f944f50db6d2ec725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32df36ed782f47d6ab892190ac369a8f" is_pre_election: true
I20260809 04:54:51.775576 31793 leader_election.cc:302] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 939f830253834cba8513a4ebc7360473, f66bd4063ca7492f944f50db6d2ec725; no voters:
I20260809 04:54:51.775676 31681 raft_consensus.cc:2454] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f66bd4063ca7492f944f50db6d2ec725 in term 0.
I20260809 04:54:51.775949 32019 raft_consensus.cc:2781] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 04:54:51.776101 32019 raft_consensus.cc:482] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 04:54:51.776218 32019 raft_consensus.cc:3037] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:51.778378 32019 raft_consensus.cc:504] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66bd4063ca7492f944f50db6d2ec725" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40855 } } peers { permanent_uuid: "939f830253834cba8513a4ebc7360473" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39989 } } peers { permanent_uuid: "32df36ed782f47d6ab892190ac369a8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 45883 } }
I20260809 04:54:51.778915 32019 leader_election.cc:288] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [CANDIDATE]: Term 1 election: Requested vote from peers 939f830253834cba8513a4ebc7360473 (127.31.93.1:39989), 32df36ed782f47d6ab892190ac369a8f (127.31.93.2:45883)
I20260809 04:54:51.779268 31439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7047cd0be0384b1f944ffcb1e1362102" candidate_uuid: "f66bd4063ca7492f944f50db6d2ec725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939f830253834cba8513a4ebc7360473"
I20260809 04:54:51.779358 31681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7047cd0be0384b1f944ffcb1e1362102" candidate_uuid: "f66bd4063ca7492f944f50db6d2ec725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32df36ed782f47d6ab892190ac369a8f"
I20260809 04:54:51.779467 31439 raft_consensus.cc:3037] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:51.779538 31681 raft_consensus.cc:3037] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:51.782575 31439 raft_consensus.cc:2454] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f66bd4063ca7492f944f50db6d2ec725 in term 1.
I20260809 04:54:51.782577 31681 raft_consensus.cc:2454] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f66bd4063ca7492f944f50db6d2ec725 in term 1.
I20260809 04:54:51.782945 31793 leader_election.cc:302] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 939f830253834cba8513a4ebc7360473, f66bd4063ca7492f944f50db6d2ec725; no voters:
I20260809 04:54:51.783200 32019 raft_consensus.cc:2781] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:54:51.783622 32019 raft_consensus.cc:686] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [term 1 LEADER]: Becoming Leader. State: Replica: f66bd4063ca7492f944f50db6d2ec725, State: Running, Role: LEADER
I20260809 04:54:51.784039 32019 consensus_queue.cc:237] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66bd4063ca7492f944f50db6d2ec725" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40855 } } peers { permanent_uuid: "939f830253834cba8513a4ebc7360473" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39989 } } peers { permanent_uuid: "32df36ed782f47d6ab892190ac369a8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 45883 } }
I20260809 04:54:51.788183 31205 catalog_manager.cc:5495] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 reported cstate change: term changed from 0 to 1, leader changed from <none> to f66bd4063ca7492f944f50db6d2ec725 (127.31.93.3). New cstate: current_term: 1 leader_uuid: "f66bd4063ca7492f944f50db6d2ec725" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66bd4063ca7492f944f50db6d2ec725" member_type: VOTER last_known_addr { host: "127.31.93.3" port: 40855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "939f830253834cba8513a4ebc7360473" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32df36ed782f47d6ab892190ac369a8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 45883 } health_report { overall_health: UNKNOWN } } }
I20260809 04:54:51.811160 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.806156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52260 (local address 127.31.93.3:40855)
0809 04:54:51.806359 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:51.806365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:51.806382 (+ 17us) server_negotiation.cc:407] Connection header received
0809 04:54:51.806569 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:51.806572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:51.806640 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:51.806761 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:54:51.807124 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.807795 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.809120 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.809515 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.809620 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.810090 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:54:51.810632 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:54:51.810758 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:51.810864 (+ 106us) server_negotiation.cc:299] Negotiation successful
0809 04:54:51.810933 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:54:51.818794 31439 raft_consensus.cc:1261] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473 [term 1 FOLLOWER]: Refusing update from remote peer f66bd4063ca7492f944f50db6d2ec725: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:54:51.818794 31681 raft_consensus.cc:1261] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f [term 1 FOLLOWER]: Refusing update from remote peer f66bd4063ca7492f944f50db6d2ec725: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:54:51.819764 32028 consensus_queue.cc:1035] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "939f830253834cba8513a4ebc7360473" member_type: VOTER last_known_addr { host: "127.31.93.1" port: 39989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:54:51.820108 32019 consensus_queue.cc:1035] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32df36ed782f47d6ab892190ac369a8f" member_type: VOTER last_known_addr { host: "127.31.93.2" port: 45883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:54:51.834131 32032 mvcc.cc:205] Tried to move back new op lower bound from 7316485291279728640 to 7316485291150950400. Current Snapshot: MvccSnapshot[applied={T|T < 7316485291279728640}]
I20260809 04:54:51.834635 32035 mvcc.cc:205] Tried to move back new op lower bound from 7316485291279728640 to 7316485291150950400. Current Snapshot: MvccSnapshot[applied={T|T < 7316485291279728640}]
I20260809 04:54:51.846958 31205 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:54:51.847182 31205 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 04:54:51.850417 31205 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3cf702f1ef56425fbc358c5471eb5e8e: Sending DeleteTablet for 3 replicas of tablet 7047cd0be0384b1f944ffcb1e1362102
I20260809 04:54:51.851475 31411 tablet_service.cc:1514] Processing DeleteTablet for tablet 7047cd0be0384b1f944ffcb1e1362102 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:51 UTC) from {username='slave'} at 127.0.0.1:43136
I20260809 04:54:51.851639 31878 tablet_service.cc:1514] Processing DeleteTablet for tablet 7047cd0be0384b1f944ffcb1e1362102 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:51 UTC) from {username='slave'} at 127.0.0.1:52252
I20260809 04:54:51.851743 31641 tablet_service.cc:1514] Processing DeleteTablet for tablet 7047cd0be0384b1f944ffcb1e1362102 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 04:54:51 UTC) from {username='slave'} at 127.0.0.1:49300
I20260809 04:54:51.853145 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 31299
I20260809 04:54:51.864110 32049 tablet_replica.cc:330] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473: stopping tablet replica
I20260809 04:54:51.864650 32049 raft_consensus.cc:2229] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:54:51.865234 32049 raft_consensus.cc:2258] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:51.867408 32049 ts_tablet_manager.cc:1905] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:54:51.874526 32049 ts_tablet_manager.cc:1918] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:54:51.874747 32049 log.cc:1196] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-0/wal/wals/7047cd0be0384b1f944ffcb1e1362102
I20260809 04:54:51.875339 32049 ts_tablet_manager.cc:1939] T 7047cd0be0384b1f944ffcb1e1362102 P 939f830253834cba8513a4ebc7360473: Deleting consensus metadata
I20260809 04:54:51.877131 31187 catalog_manager.cc:4841] TS 939f830253834cba8513a4ebc7360473 (127.31.93.1:39989): tablet 7047cd0be0384b1f944ffcb1e1362102 (table test-table [id=9c8d9a476e904acfad99d1d58fa6df7f]) successfully deleted
I20260809 04:54:51.878736 32050 tablet_replica.cc:330] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725: stopping tablet replica
I20260809 04:54:51.879206 32050 raft_consensus.cc:2229] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [term 1 LEADER]: Raft consensus shutting down.
I20260809 04:54:51.879945 32050 raft_consensus.cc:2258] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:51.882010 32050 ts_tablet_manager.cc:1905] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:54:51.885128 32051 tablet_replica.cc:330] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f: stopping tablet replica
I20260809 04:54:51.885342 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:51.854619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.31.93.1:39989)
0809 04:54:51.854789 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:51.854795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:51.877907 (+ 23112us) server_negotiation.cc:407] Connection header received
0809 04:54:51.878072 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:51.878078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:51.878168 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:51.878271 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:51.878906 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.879937 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.881673 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:51.882250 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:51.882371 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:51.883717 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:51.883727 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:51.883743 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:51.884438 (+ 695us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:51.884492 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:51.884515 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:51.884612 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:51.884900 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 04:54:51.885007 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:54:51.885535 32051 raft_consensus.cc:2229] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 04:54:51.886013 32051 raft_consensus.cc:2258] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 04:54:51.886394 31508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:51.888203 32051 ts_tablet_manager.cc:1905] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 04:54:51.891443 32050 ts_tablet_manager.cc:1918] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:54:51.891683 32050 log.cc:1196] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-2/wal/wals/7047cd0be0384b1f944ffcb1e1362102
I20260809 04:54:51.892339 32050 ts_tablet_manager.cc:1939] T 7047cd0be0384b1f944ffcb1e1362102 P f66bd4063ca7492f944f50db6d2ec725: Deleting consensus metadata
I20260809 04:54:51.894121 31181 catalog_manager.cc:4841] TS f66bd4063ca7492f944f50db6d2ec725 (127.31.93.3:40855): tablet 7047cd0be0384b1f944ffcb1e1362102 (table test-table [id=9c8d9a476e904acfad99d1d58fa6df7f]) successfully deleted
I20260809 04:54:51.894901 32051 ts_tablet_manager.cc:1918] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 04:54:51.895141 32051 log.cc:1196] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f: Deleting WAL directory at /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786250977734564-32116-0/minicluster-data/ts-1/wal/wals/7047cd0be0384b1f944ffcb1e1362102
I20260809 04:54:51.895694 32051 ts_tablet_manager.cc:1939] T 7047cd0be0384b1f944ffcb1e1362102 P 32df36ed782f47d6ab892190ac369a8f: Deleting consensus metadata
I20260809 04:54:51.897336 31181 catalog_manager.cc:4841] TS 32df36ed782f47d6ab892190ac369a8f (127.31.93.2:45883): tablet 7047cd0be0384b1f944ffcb1e1362102 (table test-table [id=9c8d9a476e904acfad99d1d58fa6df7f]) successfully deleted
I20260809 04:54:51.981526 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 31299
I20260809 04:54:52.007570 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 31527
I20260809 04:54:52.017545 31994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:52.008402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.31.93.2:45883)
0809 04:54:52.008595 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:52.008601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:52.008624 (+ 23us) server_negotiation.cc:407] Connection header received
0809 04:54:52.008695 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:52.008702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:52.008791 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:52.008874 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:52.009305 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:52.010057 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:52.011356 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:52.011761 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:52.011872 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:52.016236 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:52.016245 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:52.016260 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:52.016821 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:52.016869 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:52.016888 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:52.016987 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:52.017135 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 04:54:52.017241 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 04:54:52.018282 31752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:52.103998 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 31527
I20260809 04:54:52.124158 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 31764
I20260809 04:54:52.131249 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:52.124873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52268 (local address 127.31.93.3:40855)
0809 04:54:52.125062 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:52.125069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:52.125153 (+ 84us) server_negotiation.cc:407] Connection header received
0809 04:54:52.125357 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:52.125361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:52.125464 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:52.125553 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:52.125978 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:52.126808 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:52.128117 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:52.128594 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:52.128714 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:52.129811 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:52.129820 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:52.129833 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:52.130492 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:52.130540 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:52.130567 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:52.130651 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:52.130820 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 04:54:52.130895 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 04:54:52.132001 31959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:52.256335 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 31764
I20260809 04:54:52.283521 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 31157
I20260809 04:54:52.284539 31253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:52.389693 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 31157
2026-08-09T04:54:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2691 ms)
[----------] 6 tests from EncryptionPolicyTest (21461 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:54:52 dist-test-slave-4tdj krb5kdc[32060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:54:52 dist-test-slave-4tdj krb5kdc[32060](info): set up 2 sockets
Aug 09 04:54:52 dist-test-slave-4tdj krb5kdc[32060](info): commencing operation
krb5kdc: starting...
W20260809 04:54:53.605593 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:54:53 dist-test-slave-4tdj krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:54:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:54:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:54:53.863607 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:54.195348 32081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:54.195641 32081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:54.195786 32081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:54.202867 32081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:54:54.202975 32081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:54.203060 32081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:54.203150 32081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:54:54.203213 32081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:54:54.212839 32081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:54.214044 32081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:54.215358 32081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:54.227038 32098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:54.227115 32093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:54.227120 32096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:54.228214 32081 server_base.cc:1034] running on GCE node
I20260809 04:54:54.228976 32081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:54.230360 32081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:54.231575 32081 hybrid_clock.cc:648] HybridClock initialized: now 1786251294231523 us; error 37 us; skew 500 ppm
I20260809 04:54:54.231930 32081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:54 dist-test-slave-4tdj krb5kdc[32060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:54.234990 32081 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:54:54.236500 32081 webserver.cc:460] Webserver started at http://127.0.0.1:37397/ using document root <none> and password file <none>
I20260809 04:54:54.237154 32081 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:54.237286 32081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:54.237599 32081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:54.240604 32081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "9d44cf4c6b7c423fa5f37ae6bc6d07ab"
format_stamp: "Formatted at 2026-08-09 04:54:54 on dist-test-slave-4tdj"
server_key: "b7ab45d3e441dc5fb9bf460d12924e49"
server_key_iv: "c120fff8cf8c104fee0d0bee073b47e4"
server_key_version: "encryptionkey@0"
I20260809 04:54:54.241389 32081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "9d44cf4c6b7c423fa5f37ae6bc6d07ab"
format_stamp: "Formatted at 2026-08-09 04:54:54 on dist-test-slave-4tdj"
server_key: "b7ab45d3e441dc5fb9bf460d12924e49"
server_key_iv: "c120fff8cf8c104fee0d0bee073b47e4"
server_key_version: "encryptionkey@0"
I20260809 04:54:54.246482 32081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260809 04:54:54.249877 32109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:54.251154 32081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:54:54.251331 32081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "9d44cf4c6b7c423fa5f37ae6bc6d07ab"
format_stamp: "Formatted at 2026-08-09 04:54:54 on dist-test-slave-4tdj"
server_key: "b7ab45d3e441dc5fb9bf460d12924e49"
server_key_iv: "c120fff8cf8c104fee0d0bee073b47e4"
server_key_version: "encryptionkey@0"
I20260809 04:54:54.251521 32081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:54.281582 32081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:54.285568 32081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:54.285849 32081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:54.304260 32081 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38781
I20260809 04:54:54.304279 32200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38781 every 8 connection(s)
I20260809 04:54:54.305661 32081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:54:54.308053 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 32081
I20260809 04:54:54.308274 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:54:54.309186 32116 external_mini_cluster.cc:1428] Setting key 9d816ff9ce6bf67593956c2738b86463
I20260809 04:54:54.309911 32201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:54.323669 32201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab: Bootstrap starting.
Aug 09 04:54:54 dist-test-slave-4tdj krb5kdc[32060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 04:54:54.327193 32201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:54.328238 32201 log.cc:824] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:54.331526 32201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab: No bootstrap required, opened a new log
I20260809 04:54:54.333922 32206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:54.311161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33563 (local address 127.0.0.1:38781)
0809 04:54:54.311968 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:54.311986 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:54.312029 (+ 43us) server_negotiation.cc:407] Connection header received
0809 04:54:54.314457 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:54.314482 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:54.314900 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:54.315254 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:54.316998 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:54.318280 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:54.320209 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:54.320896 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:54.321042 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:54.326030 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:54.326038 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:54.326052 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:54.326102 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:54.329384 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:54.330125 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:54.330133 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:54.330141 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:54.330377 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:54.330815 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:54.330820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:54.330823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:54.331506 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:54.331804 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:54.332303 (+ 499us) server_negotiation.cc:299] Negotiation successful
0809 04:54:54.332624 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":359,"threads_started":1}
I20260809 04:54:54.337904 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:54:54.340391 32201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d44cf4c6b7c423fa5f37ae6bc6d07ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } }
I20260809 04:54:54.340730 32201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:54.340829 32201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d44cf4c6b7c423fa5f37ae6bc6d07ab, State: Initialized, Role: FOLLOWER
I20260809 04:54:54.341427 32201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d44cf4c6b7c423fa5f37ae6bc6d07ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } }
I20260809 04:54:54.341660 32201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:54:54.341782 32201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:54:54.341936 32201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:54.344422 32201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d44cf4c6b7c423fa5f37ae6bc6d07ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } }
I20260809 04:54:54.344956 32201 leader_election.cc:302] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d44cf4c6b7c423fa5f37ae6bc6d07ab; no voters:
I20260809 04:54:54.345641 32201 leader_election.cc:288] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:54:54.345736 32219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:54:54.348070 32219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [term 1 LEADER]: Becoming Leader. State: Replica: 9d44cf4c6b7c423fa5f37ae6bc6d07ab, State: Running, Role: LEADER
I20260809 04:54:54.348556 32219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d44cf4c6b7c423fa5f37ae6bc6d07ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } }
I20260809 04:54:54.349124 32201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:54:54.351002 32221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d44cf4c6b7c423fa5f37ae6bc6d07ab. Latest consensus state: current_term: 1 leader_uuid: "9d44cf4c6b7c423fa5f37ae6bc6d07ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d44cf4c6b7c423fa5f37ae6bc6d07ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } } }
I20260809 04:54:54.351325 32221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:54.351902 32220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d44cf4c6b7c423fa5f37ae6bc6d07ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d44cf4c6b7c423fa5f37ae6bc6d07ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } } }
I20260809 04:54:54.352532 32220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:54.359830 32206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:54.342576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56837 (local address 127.0.0.1:38781)
0809 04:54:54.342745 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:54.342751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:54.343220 (+ 469us) server_negotiation.cc:407] Connection header received
0809 04:54:54.343343 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:54.343348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:54.343424 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:54.343520 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:54.344895 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:54.346036 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:54.347273 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:54.347955 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:54.348114 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:54.351346 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:54.351353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:54.351358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:54:54.351394 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:54.355383 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:54.356539 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:54.356544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:54.356547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:54.356694 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:54:54.357367 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:54:54.357372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:54:54.357376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:54:54.357818 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:54.359112 (+ 1294us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:54.359292 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 04:54:54.359463 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:54:54.360095 32230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:54:54.365188 32230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:54:54.373643 32230 catalog_manager.cc:1347] Generated new cluster ID: 4c137870051f492ea8de5a98f0a4eed7
I20260809 04:54:54.373744 32230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:54:54.395627 32230 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:54:54.396816 32230 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:54:54.404760 32230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d44cf4c6b7c423fa5f37ae6bc6d07ab: Generated new TSK 0
I20260809 04:54:54.405514 32230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:54:54.423938 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 32081
I20260809 04:54:54.425606 32191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:54:54.514773 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 32081
W20260809 04:54:54.538098 32058 connection.cc:535] client connection to 127.0.0.1:38781 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38781): Connection reset by peer (error 104)
2026-08-09T04:54:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2121 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:54:54 dist-test-slave-4tdj krb5kdc[32255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:54:54 dist-test-slave-4tdj krb5kdc[32255](info): set up 2 sockets
Aug 09 04:54:54 dist-test-slave-4tdj krb5kdc[32255](info): commencing operation
krb5kdc: starting...
W20260809 04:54:55.725883 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.001s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:54:55 dist-test-slave-4tdj krb5kdc[32255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:54:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:54:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:54:55.970760 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:56.287495 32273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:56.287773 32273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:56.287870 32273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:56.294885 32273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:54:56.294996 32273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:56.295040 32273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:56.295106 32273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:54:56.295149 32273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:54:56.304627 32273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33431
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:56.305823 32273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:56.307078 32273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:56.318320 32280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:56.318235 32281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:56.318279 32285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:56.319105 32273 server_base.cc:1034] running on GCE node
I20260809 04:54:56.319851 32273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:56.321171 32273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:56.322386 32273 hybrid_clock.cc:648] HybridClock initialized: now 1786251296322344 us; error 29 us; skew 500 ppm
I20260809 04:54:56.322722 32273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:56 dist-test-slave-4tdj krb5kdc[32255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:56.325804 32273 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:54:56.327180 32273 webserver.cc:460] Webserver started at http://127.0.0.1:42777/ using document root <none> and password file <none>
I20260809 04:54:56.327806 32273 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:56.327924 32273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:56.328204 32273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:56.331110 32273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "d00236e1b78341aebe80d2e64380259d"
format_stamp: "Formatted at 2026-08-09 04:54:56 on dist-test-slave-4tdj"
server_key: "7b07912a846e802070e533ef1c6009b6"
server_key_iv: "bdda4ec8d0b7763eb2c38ffd4158239c"
server_key_version: "encryptionkey@0"
I20260809 04:54:56.331826 32273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "d00236e1b78341aebe80d2e64380259d"
format_stamp: "Formatted at 2026-08-09 04:54:56 on dist-test-slave-4tdj"
server_key: "7b07912a846e802070e533ef1c6009b6"
server_key_iv: "bdda4ec8d0b7763eb2c38ffd4158239c"
server_key_version: "encryptionkey@0"
I20260809 04:54:56.336721 32273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260809 04:54:56.340090 32293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:56.341316 32273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 04:54:56.341471 32273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "d00236e1b78341aebe80d2e64380259d"
format_stamp: "Formatted at 2026-08-09 04:54:56 on dist-test-slave-4tdj"
server_key: "7b07912a846e802070e533ef1c6009b6"
server_key_iv: "bdda4ec8d0b7763eb2c38ffd4158239c"
server_key_version: "encryptionkey@0"
I20260809 04:54:56.341638 32273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:56.354341 32273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:56.357784 32273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:56.358016 32273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:56.374722 32273 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33431
I20260809 04:54:56.374775 32385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33431 every 8 connection(s)
I20260809 04:54:56.375964 32273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:54:56.379575 32388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:56.380522 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 32273
I20260809 04:54:56.380705 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:54:56.381209 32116 external_mini_cluster.cc:1428] Setting key 512dbb00ae44aa0a5acf19c5364a239c
I20260809 04:54:56.393356 32388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d: Bootstrap starting.
I20260809 04:54:56.395182 32391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:56.382931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39799 (local address 127.0.0.1:33431)
0809 04:54:56.383743 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:56.383762 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:56.383815 (+ 53us) server_negotiation.cc:407] Connection header received
0809 04:54:56.385726 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:56.385754 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:56.386237 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:56.386682 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:56.387161 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:56.388946 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:56.390216 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:56.391266 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:56.391440 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:56.392531 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:56.392540 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:56.392558 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:56.393226 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:56.393322 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:56.393358 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:56.393510 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:56.393754 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 04:54:56.393921 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":308,"threads_started":1}
I20260809 04:54:56.396822 32388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:56.397927 32388 log.cc:824] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:56.398747 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:54:56.400830 32388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d: No bootstrap required, opened a new log
I20260809 04:54:56.410030 32391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:56.403795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44083 (local address 127.0.0.1:33431)
0809 04:54:56.403976 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:56.403981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:56.404306 (+ 325us) server_negotiation.cc:407] Connection header received
0809 04:54:56.404445 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:56.404449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:56.404508 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:56.404585 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:56.405156 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:56.405889 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:56.407735 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:54:56.408183 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:54:56.408282 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:54:56.409366 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:56.409374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:56.409378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:56.409398 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:56.409448 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:56.409463 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:56.409542 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:56.409716 (+ 174us) server_negotiation.cc:299] Negotiation successful
0809 04:54:56.409754 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 04:54:56.410413 32388 raft_consensus.cc:348] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00236e1b78341aebe80d2e64380259d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33431 } }
I20260809 04:54:56.410851 32388 raft_consensus.cc:374] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:56.410948 32388 raft_consensus.cc:729] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00236e1b78341aebe80d2e64380259d, State: Initialized, Role: FOLLOWER
I20260809 04:54:56.411549 32388 consensus_queue.cc:260] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00236e1b78341aebe80d2e64380259d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33431 } }
I20260809 04:54:56.411770 32388 raft_consensus.cc:388] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:54:56.411900 32388 raft_consensus.cc:482] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:54:56.412051 32388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:56.414054 32388 raft_consensus.cc:504] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00236e1b78341aebe80d2e64380259d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33431 } }
I20260809 04:54:56.414552 32388 leader_election.cc:302] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d00236e1b78341aebe80d2e64380259d; no voters:
I20260809 04:54:56.415076 32388 leader_election.cc:288] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:54:56.415187 32406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:54:56.415817 32406 raft_consensus.cc:686] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [term 1 LEADER]: Becoming Leader. State: Replica: d00236e1b78341aebe80d2e64380259d, State: Running, Role: LEADER
I20260809 04:54:56.416366 32406 consensus_queue.cc:237] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00236e1b78341aebe80d2e64380259d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33431 } }
I20260809 04:54:56.416918 32388 sys_catalog.cc:564] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:54:56.420125 32407 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d00236e1b78341aebe80d2e64380259d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00236e1b78341aebe80d2e64380259d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33431 } } }
I20260809 04:54:56.420151 32409 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d00236e1b78341aebe80d2e64380259d. Latest consensus state: current_term: 1 leader_uuid: "d00236e1b78341aebe80d2e64380259d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00236e1b78341aebe80d2e64380259d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33431 } } }
I20260809 04:54:56.420537 32407 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:56.420538 32409 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:56.422081 32419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:54:56.426190 32419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:54:56.433516 32419 catalog_manager.cc:1347] Generated new cluster ID: dc6a309979c14e9eb2bdeffdd26029d8
I20260809 04:54:56.433622 32419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:54:56.464287 32419 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:54:56.466048 32419 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:54:56.476265 32419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d00236e1b78341aebe80d2e64380259d: Generated new TSK 0
I20260809 04:54:56.477216 32419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:54:56.494006 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 32273
W20260809 04:54:56.495730 32378 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39799
W20260809 04:54:56.496397 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 32273: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:54:56.496592 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 32273
W20260809 04:54:56.520705 32253 connection.cc:535] client connection to 127.0.0.1:33431 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33431): Connection reset by peer (error 104)
2026-08-09T04:54:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1982 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:54:56 dist-test-slave-4tdj krb5kdc[32447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:54:56 dist-test-slave-4tdj krb5kdc[32447](info): set up 2 sockets
Aug 09 04:54:56 dist-test-slave-4tdj krb5kdc[32447](info): commencing operation
krb5kdc: starting...
W20260809 04:54:57.703895 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:54:57 dist-test-slave-4tdj krb5kdc[32447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:54:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:54:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:54:57.967291 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:54:58.282513 32477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:54:58.282770 32477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:54:58.282895 32477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:54:58.289801 32477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:54:58.289898 32477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:54:58.289974 32477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:54:58.290060 32477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:54:58.290119 32477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:54:58.299394 32477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42509
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:54:58.300540 32477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:54:58.301712 32477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:54:58.312606 32486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:58.312731 32489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:54:58.312777 32487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:54:58.313634 32477 server_base.cc:1034] running on GCE node
I20260809 04:54:58.314355 32477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:54:58.315582 32477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:54:58.316769 32477 hybrid_clock.cc:648] HybridClock initialized: now 1786251298316720 us; error 39 us; skew 500 ppm
I20260809 04:54:58.317075 32477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:54:58 dist-test-slave-4tdj krb5kdc[32447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:54:58.322415 32477 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:54:58.323701 32477 webserver.cc:460] Webserver started at http://127.0.0.1:43833/ using document root <none> and password file <none>
I20260809 04:54:58.324282 32477 fs_manager.cc:359] Metadata directory not provided
I20260809 04:54:58.324378 32477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:54:58.324625 32477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:54:58.327425 32477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "410585d592c14a70991923ce002a1dbd"
format_stamp: "Formatted at 2026-08-09 04:54:58 on dist-test-slave-4tdj"
server_key: "cc8eb786e9525ea48257d9c17ec392d0"
server_key_iv: "835a86c62ca1c11f6109255868b2b63e"
server_key_version: "encryptionkey@0"
I20260809 04:54:58.328104 32477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "410585d592c14a70991923ce002a1dbd"
format_stamp: "Formatted at 2026-08-09 04:54:58 on dist-test-slave-4tdj"
server_key: "cc8eb786e9525ea48257d9c17ec392d0"
server_key_iv: "835a86c62ca1c11f6109255868b2b63e"
server_key_version: "encryptionkey@0"
I20260809 04:54:58.332791 32477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:54:58.335917 32503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:54:58.337149 32477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:54:58.337322 32477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "410585d592c14a70991923ce002a1dbd"
format_stamp: "Formatted at 2026-08-09 04:54:58 on dist-test-slave-4tdj"
server_key: "cc8eb786e9525ea48257d9c17ec392d0"
server_key_iv: "835a86c62ca1c11f6109255868b2b63e"
server_key_version: "encryptionkey@0"
I20260809 04:54:58.337502 32477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:54:58.342629 32477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:54:58.346447 32477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:54:58.346706 32477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:54:58.362741 32477 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42509
I20260809 04:54:58.362804 32600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42509 every 8 connection(s)
I20260809 04:54:58.363960 32477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:54:58.367411 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 32477
I20260809 04:54:58.367514 32601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:54:58.367607 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:54:58.368160 32116 external_mini_cluster.cc:1428] Setting key e6a49dacc378748ea87df3eb54e9b8fa
I20260809 04:54:58.375962 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:58.369756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42071 (local address 127.0.0.1:42509)
0809 04:54:58.370575 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:58.370593 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:58.370640 (+ 47us) server_negotiation.cc:407] Connection header received
0809 04:54:58.372350 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:58.372375 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:58.372794 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:58.373190 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:58.374140 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:58.374148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:58.374168 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:58.374188 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:58.374251 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:58.374285 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:58.374530 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:58.374667 (+ 137us) server_negotiation.cc:299] Negotiation successful
0809 04:54:58.374847 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":312,"threads_started":1}
I20260809 04:54:58.379206 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:54:58.381630 32601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd: Bootstrap starting.
I20260809 04:54:58.385304 32601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd: Neither blocks nor log segments found. Creating new log.
I20260809 04:54:58.386583 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:54:58.384239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45089 (local address 127.0.0.1:42509)
0809 04:54:58.384386 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 04:54:58.384391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:54:58.384874 (+ 483us) server_negotiation.cc:407] Connection header received
0809 04:54:58.385024 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:54:58.385028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:54:58.385078 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:54:58.385147 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:54:58.385971 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:54:58.385975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:54:58.385978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:54:58.385992 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:54:58.386026 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:54:58.386040 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:54:58.386128 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:54:58.386260 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 04:54:58.386325 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 04:54:58.386592 32601 log.cc:824] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd: Log is configured to *not* fsync() on all Append() calls
I20260809 04:54:58.389328 32601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd: No bootstrap required, opened a new log
I20260809 04:54:58.396209 32601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410585d592c14a70991923ce002a1dbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42509 } }
I20260809 04:54:58.396512 32601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:54:58.396581 32601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410585d592c14a70991923ce002a1dbd, State: Initialized, Role: FOLLOWER
I20260809 04:54:58.397080 32601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410585d592c14a70991923ce002a1dbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42509 } }
I20260809 04:54:58.397298 32601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:54:58.397395 32601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:54:58.397512 32601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:54:58.399504 32601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410585d592c14a70991923ce002a1dbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42509 } }
I20260809 04:54:58.400019 32601 leader_election.cc:302] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 410585d592c14a70991923ce002a1dbd; no voters:
I20260809 04:54:58.400556 32601 leader_election.cc:288] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:54:58.400676 32614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:54:58.401268 32614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [term 1 LEADER]: Becoming Leader. State: Replica: 410585d592c14a70991923ce002a1dbd, State: Running, Role: LEADER
I20260809 04:54:58.401746 32614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410585d592c14a70991923ce002a1dbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42509 } }
I20260809 04:54:58.402390 32601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:54:58.404510 32617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 410585d592c14a70991923ce002a1dbd. Latest consensus state: current_term: 1 leader_uuid: "410585d592c14a70991923ce002a1dbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410585d592c14a70991923ce002a1dbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42509 } } }
I20260809 04:54:58.404590 32615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "410585d592c14a70991923ce002a1dbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410585d592c14a70991923ce002a1dbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42509 } } }
I20260809 04:54:58.404889 32617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:58.404907 32615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd [sys.catalog]: This master's current role is: LEADER
I20260809 04:54:58.409245 32625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:54:58.415784 32625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:54:58.424008 32625 catalog_manager.cc:1347] Generated new cluster ID: 0678ed4667204a559bb4796595e3774f
I20260809 04:54:58.424115 32625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:54:58.451493 32625 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:54:58.451866 32625 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:54:58.459597 32625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 410585d592c14a70991923ce002a1dbd: Generated new TSK 0
I20260809 04:54:58.460269 32625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:54:58.470744 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 32477
W20260809 04:54:58.471657 32595 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42071
W20260809 04:54:58.472288 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 32477: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:54:58.472411 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 32477
2026-08-09T04:54:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1978 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:54:58 dist-test-slave-4tdj krb5kdc[32650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:54:58 dist-test-slave-4tdj krb5kdc[32650](info): set up 2 sockets
Aug 09 04:54:58 dist-test-slave-4tdj krb5kdc[32650](info): commencing operation
krb5kdc: starting...
W20260809 04:54:59.688665 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.002s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:54:59 dist-test-slave-4tdj krb5kdc[32650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:54:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:54:59.926584 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:00.254925 32670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:00.255205 32670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:00.255338 32670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:00.262341 32670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:00.262447 32670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:00.262524 32670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:00.262615 32670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:00.262686 32670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:00.272084 32670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:00.273294 32670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:00.274580 32670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:00.285632 32680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:00.285590 32688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:00.285676 32681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:00.286705 32670 server_base.cc:1034] running on GCE node
I20260809 04:55:00.287531 32670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:00.288973 32670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:00.290170 32670 hybrid_clock.cc:648] HybridClock initialized: now 1786251300290117 us; error 46 us; skew 500 ppm
I20260809 04:55:00.290565 32670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:00 dist-test-slave-4tdj krb5kdc[32650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:00.293577 32670 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:55:00.294941 32670 webserver.cc:460] Webserver started at http://127.0.0.1:42317/ using document root <none> and password file <none>
I20260809 04:55:00.295569 32670 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:00.295686 32670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:00.295955 32670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:00.298847 32670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "0e867c8fb17f4388b6542ec0f0af793a"
format_stamp: "Formatted at 2026-08-09 04:55:00 on dist-test-slave-4tdj"
server_key: "030568147b6c0d9234b20a0a06a5f407"
server_key_iv: "4f20805d4b9766499362b6029c1eb7b4"
server_key_version: "encryptionkey@0"
I20260809 04:55:00.299580 32670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "0e867c8fb17f4388b6542ec0f0af793a"
format_stamp: "Formatted at 2026-08-09 04:55:00 on dist-test-slave-4tdj"
server_key: "030568147b6c0d9234b20a0a06a5f407"
server_key_iv: "4f20805d4b9766499362b6029c1eb7b4"
server_key_version: "encryptionkey@0"
I20260809 04:55:00.304371 32670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:55:00.307626 32700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:00.308847 32670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:55:00.309007 32670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "0e867c8fb17f4388b6542ec0f0af793a"
format_stamp: "Formatted at 2026-08-09 04:55:00 on dist-test-slave-4tdj"
server_key: "030568147b6c0d9234b20a0a06a5f407"
server_key_iv: "4f20805d4b9766499362b6029c1eb7b4"
server_key_version: "encryptionkey@0"
I20260809 04:55:00.309170 32670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:00.331454 32670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:00.335215 32670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:00.335466 32670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:00.352667 32670 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35401
I20260809 04:55:00.352703 323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35401 every 8 connection(s)
I20260809 04:55:00.353971 32670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:00.357568 324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:00.358390 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 32670
I20260809 04:55:00.358569 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:00.358997 32116 external_mini_cluster.cc:1428] Setting key 292f423e514627b81e9820202c8fde2d
I20260809 04:55:00.371444 324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a: Bootstrap starting.
Aug 09 04:55:00 dist-test-slave-4tdj krb5kdc[32650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 04:55:00.374791 324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:00.375960 324 log.cc:824] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:00.378924 324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a: No bootstrap required, opened a new log
I20260809 04:55:00.381976 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:00.360807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54945 (local address 127.0.0.1:35401)
0809 04:55:00.361582 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:00.361596 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:00.361646 (+ 50us) server_negotiation.cc:407] Connection header received
0809 04:55:00.363925 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:00.363955 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:00.364467 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:00.364911 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:00.366196 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:00.368175 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:00.369436 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:00.370461 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:00.370625 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:00.375005 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:00.375014 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:00.375027 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:00.375069 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:00.377959 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:00.378847 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:00.378852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:00.378858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:00.379057 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:00.379501 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:00.379504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:00.379506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:00.379978 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:00.380190 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:00.380663 (+ 473us) server_negotiation.cc:299] Negotiation successful
0809 04:55:00.380992 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":323,"threads_started":1}
I20260809 04:55:00.385365 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:00.386905 324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e867c8fb17f4388b6542ec0f0af793a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35401 } }
I20260809 04:55:00.387208 324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:55:00.387307 324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e867c8fb17f4388b6542ec0f0af793a, State: Initialized, Role: FOLLOWER
I20260809 04:55:00.387877 324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e867c8fb17f4388b6542ec0f0af793a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35401 } }
I20260809 04:55:00.388100 324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:55:00.388288 324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:55:00.388473 324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:00.391079 324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e867c8fb17f4388b6542ec0f0af793a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35401 } }
I20260809 04:55:00.391530 324 leader_election.cc:302] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e867c8fb17f4388b6542ec0f0af793a; no voters:
I20260809 04:55:00.392160 324 leader_election.cc:288] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:55:00.392251 337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:00.393380 337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [term 1 LEADER]: Becoming Leader. State: Replica: 0e867c8fb17f4388b6542ec0f0af793a, State: Running, Role: LEADER
I20260809 04:55:00.393880 337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e867c8fb17f4388b6542ec0f0af793a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35401 } }
I20260809 04:55:00.394270 324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:00.396253 340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e867c8fb17f4388b6542ec0f0af793a. Latest consensus state: current_term: 1 leader_uuid: "0e867c8fb17f4388b6542ec0f0af793a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e867c8fb17f4388b6542ec0f0af793a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35401 } } }
I20260809 04:55:00.396622 340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:00.396916 338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e867c8fb17f4388b6542ec0f0af793a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e867c8fb17f4388b6542ec0f0af793a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35401 } } }
I20260809 04:55:00.397182 338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:00.406330 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:00.389517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35959 (local address 127.0.0.1:35401)
0809 04:55:00.389677 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:00.389683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:00.390059 (+ 376us) server_negotiation.cc:407] Connection header received
0809 04:55:00.390200 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:00.390206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:00.390266 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:00.390371 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:00.391777 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:00.392924 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:00.394728 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:00.395423 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:00.396184 (+ 761us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:00.398576 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:00.398583 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:00.398588 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:00.398627 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:00.402211 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:00.403138 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:00.403143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:00.403148 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:00.403305 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:00.403758 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:00.403763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:00.403767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:00.404170 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:00.405534 (+ 1364us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:00.405690 (+ 156us) server_negotiation.cc:299] Negotiation successful
0809 04:55:00.405909 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 04:55:00.409078 347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:00.414060 347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:55:00.422822 347 catalog_manager.cc:1347] Generated new cluster ID: 987d9219bba34804b209e8558cac8f93
I20260809 04:55:00.422927 347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:00.435518 347 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:00.437268 347 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:00.448539 347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e867c8fb17f4388b6542ec0f0af793a: Generated new TSK 0
I20260809 04:55:00.449328 347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:00.454401 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 32670
I20260809 04:55:00.455474 312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:55:00.541178 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 32670
2026-08-09T04:55:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2066 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:55:00 dist-test-slave-4tdj krb5kdc[380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:55:00 dist-test-slave-4tdj krb5kdc[380](info): set up 2 sockets
Aug 09 04:55:00 dist-test-slave-4tdj krb5kdc[380](info): commencing operation
krb5kdc: starting...
W20260809 04:55:01.743510 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:55:01 dist-test-slave-4tdj krb5kdc[380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:55:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:55:02.015497 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44373
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:02.337271 402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:02.337540 402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:02.337675 402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:02.344623 402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:02.344727 402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:02.344800 402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:02.344879 402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:02.344934 402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:02.354378 402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:02.355559 402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:02.356884 402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:02.367959 410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:02.368017 411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:02.368050 414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:02.368773 402 server_base.cc:1034] running on GCE node
I20260809 04:55:02.369565 402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:02.370920 402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:02.372116 402 hybrid_clock.cc:648] HybridClock initialized: now 1786251302372069 us; error 36 us; skew 500 ppm
I20260809 04:55:02.372433 402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:02 dist-test-slave-4tdj krb5kdc[380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:02.375384 402 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:55:02.376740 402 webserver.cc:460] Webserver started at http://127.0.0.1:44883/ using document root <none> and password file <none>
I20260809 04:55:02.377357 402 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:02.377475 402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:02.377765 402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:02.380686 402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "adff423e84064d2eb563227b633b1043"
format_stamp: "Formatted at 2026-08-09 04:55:02 on dist-test-slave-4tdj"
server_key: "f8418aa63fc44ded1915956f99f5220f"
server_key_iv: "dff771f4b77a3ed9a10fa53a5a8793d4"
server_key_version: "encryptionkey@0"
I20260809 04:55:02.381408 402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "adff423e84064d2eb563227b633b1043"
format_stamp: "Formatted at 2026-08-09 04:55:02 on dist-test-slave-4tdj"
server_key: "f8418aa63fc44ded1915956f99f5220f"
server_key_iv: "dff771f4b77a3ed9a10fa53a5a8793d4"
server_key_version: "encryptionkey@0"
I20260809 04:55:02.386371 402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 04:55:02.389567 429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:02.390923 402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:55:02.391094 402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "adff423e84064d2eb563227b633b1043"
format_stamp: "Formatted at 2026-08-09 04:55:02 on dist-test-slave-4tdj"
server_key: "f8418aa63fc44ded1915956f99f5220f"
server_key_iv: "dff771f4b77a3ed9a10fa53a5a8793d4"
server_key_version: "encryptionkey@0"
I20260809 04:55:02.391281 402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:02.409456 402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:02.413383 402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:02.413664 402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:02.430806 402 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44373
I20260809 04:55:02.430840 512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44373 every 8 connection(s)
I20260809 04:55:02.432081 402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:02.435637 513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:02.438519 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 402
I20260809 04:55:02.438699 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:02.439257 32116 external_mini_cluster.cc:1428] Setting key d26ba08c15ee67c7333fbf45b3df0825
I20260809 04:55:02.447448 513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043: Bootstrap starting.
I20260809 04:55:02.450949 513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:02.452023 518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:02.441053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50459 (local address 127.0.0.1:44373)
0809 04:55:02.441794 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:02.441806 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:02.441847 (+ 41us) server_negotiation.cc:407] Connection header received
0809 04:55:02.443503 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:02.443533 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:02.443904 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:02.444213 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:02.444807 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:02.446121 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:02.447917 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:02.448566 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:02.448685 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:02.449892 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:02.449899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:02.449911 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:02.450437 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:02.450519 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:02.450543 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:02.450631 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:02.450845 (+ 214us) server_negotiation.cc:299] Negotiation successful
0809 04:55:02.451070 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":326,"threads_started":1}
I20260809 04:55:02.452047 513 log.cc:824] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:02.454865 513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043: No bootstrap required, opened a new log
I20260809 04:55:02.455444 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:02.462553 513 raft_consensus.cc:348] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adff423e84064d2eb563227b633b1043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } }
I20260809 04:55:02.462914 513 raft_consensus.cc:374] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:55:02.463027 513 raft_consensus.cc:729] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adff423e84064d2eb563227b633b1043, State: Initialized, Role: FOLLOWER
I20260809 04:55:02.463598 513 consensus_queue.cc:260] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adff423e84064d2eb563227b633b1043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } }
I20260809 04:55:02.463833 513 raft_consensus.cc:388] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:55:02.463963 513 raft_consensus.cc:482] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:55:02.464105 513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:02.465185 518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:02.459497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51911 (local address 127.0.0.1:44373)
0809 04:55:02.459655 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:02.459659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:02.460075 (+ 416us) server_negotiation.cc:407] Connection header received
0809 04:55:02.460243 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:02.460246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:02.460300 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:02.460376 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:02.460874 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:02.461588 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:02.462914 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:02.463386 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:02.463492 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:02.464534 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:02.464539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:02.464542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:02.464557 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:02.464595 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:02.464609 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:02.464678 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:02.464826 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 04:55:02.464892 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 04:55:02.466773 513 raft_consensus.cc:504] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adff423e84064d2eb563227b633b1043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } }
I20260809 04:55:02.467240 513 leader_election.cc:302] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adff423e84064d2eb563227b633b1043; no voters:
I20260809 04:55:02.467748 513 leader_election.cc:288] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:55:02.467842 530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:02.468432 530 raft_consensus.cc:686] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [term 1 LEADER]: Becoming Leader. State: Replica: adff423e84064d2eb563227b633b1043, State: Running, Role: LEADER
I20260809 04:55:02.468961 530 consensus_queue.cc:237] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adff423e84064d2eb563227b633b1043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } }
I20260809 04:55:02.469617 513 sys_catalog.cc:564] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:02.471544 536 sys_catalog.cc:455] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adff423e84064d2eb563227b633b1043. Latest consensus state: current_term: 1 leader_uuid: "adff423e84064d2eb563227b633b1043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adff423e84064d2eb563227b633b1043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } } }
I20260809 04:55:02.471889 536 sys_catalog.cc:458] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:02.472558 533 sys_catalog.cc:455] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adff423e84064d2eb563227b633b1043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adff423e84064d2eb563227b633b1043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44373 } } }
I20260809 04:55:02.472891 533 sys_catalog.cc:458] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:02.478538 542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:02.482455 542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:55:02.490130 542 catalog_manager.cc:1347] Generated new cluster ID: 0e2c2a8b43ca4fbf878c8966c7a00297
I20260809 04:55:02.490223 542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:02.506752 542 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:02.507915 542 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:02.513100 542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adff423e84064d2eb563227b633b1043: Generated new TSK 0
I20260809 04:55:02.513763 542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:02.522416 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 402
W20260809 04:55:02.523406 506 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50459
W20260809 04:55:02.524085 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 402: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:55:02.524215 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 402
2026-08-09T04:55:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1983 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:55:02 dist-test-slave-4tdj krb5kdc[565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:55:02 dist-test-slave-4tdj krb5kdc[565](info): set up 2 sockets
Aug 09 04:55:02 dist-test-slave-4tdj krb5kdc[565](info): commencing operation
krb5kdc: starting...
W20260809 04:55:03.740875 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.005s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:55:03 dist-test-slave-4tdj krb5kdc[565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:55:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:55:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:55:03.978591 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:04.298739 583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:04.299010 583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:04.299150 583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:04.306010 583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:04.306116 583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:04.306192 583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:04.306274 583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:04.306358 583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:04.315587 583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:04.316723 583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:04.317927 583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:04.328684 588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:04.328717 591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:04.329057 594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:04.329664 583 server_base.cc:1034] running on GCE node
I20260809 04:55:04.330446 583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:04.331688 583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:04.332888 583 hybrid_clock.cc:648] HybridClock initialized: now 1786251304332849 us; error 29 us; skew 500 ppm
I20260809 04:55:04.333207 583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:04 dist-test-slave-4tdj krb5kdc[565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:04.341503 583 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:55:04.342947 583 webserver.cc:460] Webserver started at http://127.0.0.1:34903/ using document root <none> and password file <none>
I20260809 04:55:04.343576 583 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:04.343693 583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:04.343971 583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:04.346855 583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "7338f5d26ea14d58a526a8da5784779e"
format_stamp: "Formatted at 2026-08-09 04:55:04 on dist-test-slave-4tdj"
server_key: "9bc0b1059637706bd561830d32238418"
server_key_iv: "cc0aeec1deea1d2627cb4518a1e60283"
server_key_version: "encryptionkey@0"
I20260809 04:55:04.347581 583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "7338f5d26ea14d58a526a8da5784779e"
format_stamp: "Formatted at 2026-08-09 04:55:04 on dist-test-slave-4tdj"
server_key: "9bc0b1059637706bd561830d32238418"
server_key_iv: "cc0aeec1deea1d2627cb4518a1e60283"
server_key_version: "encryptionkey@0"
I20260809 04:55:04.352438 583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260809 04:55:04.355628 604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:04.356798 583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:55:04.356958 583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "7338f5d26ea14d58a526a8da5784779e"
format_stamp: "Formatted at 2026-08-09 04:55:04 on dist-test-slave-4tdj"
server_key: "9bc0b1059637706bd561830d32238418"
server_key_iv: "cc0aeec1deea1d2627cb4518a1e60283"
server_key_version: "encryptionkey@0"
I20260809 04:55:04.357133 583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:04.362545 583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:04.366077 583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:04.366364 583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:04.383235 583 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39179
I20260809 04:55:04.383288 694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39179 every 8 connection(s)
I20260809 04:55:04.384477 583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:04.387952 696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:04.389508 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 583
I20260809 04:55:04.389670 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:04.390100 32116 external_mini_cluster.cc:1428] Setting key b1ea9b2fbc1d5a41ff4ba9271809ae32
I20260809 04:55:04.397150 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:04.391718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59933 (local address 127.0.0.1:39179)
0809 04:55:04.392486 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:04.392500 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:04.392541 (+ 41us) server_negotiation.cc:407] Connection header received
0809 04:55:04.394088 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:04.394112 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:04.394463 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:04.394777 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:04.395709 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:04.395715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:04.395727 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:04.395740 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:04.395791 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:04.395813 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:04.395972 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:04.396067 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 04:55:04.396209 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":348,"threads_started":1}
I20260809 04:55:04.400455 696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e: Bootstrap starting.
I20260809 04:55:04.400493 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:04.404147 696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:04.405404 696 log.cc:824] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:04.407727 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:04.405547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47679 (local address 127.0.0.1:39179)
0809 04:55:04.405715 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:04.405720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:04.406086 (+ 366us) server_negotiation.cc:407] Connection header received
0809 04:55:04.406245 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:04.406250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:04.406323 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:04.406444 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:04.407177 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:04.407181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:04.407184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:04.407198 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:04.407234 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:04.407247 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:04.407322 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:04.407430 (+ 108us) server_negotiation.cc:299] Negotiation successful
0809 04:55:04.407471 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:55:04.408442 696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e: No bootstrap required, opened a new log
I20260809 04:55:04.415282 696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338f5d26ea14d58a526a8da5784779e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39179 } }
I20260809 04:55:04.415621 696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:55:04.415707 696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7338f5d26ea14d58a526a8da5784779e, State: Initialized, Role: FOLLOWER
I20260809 04:55:04.416201 696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338f5d26ea14d58a526a8da5784779e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39179 } }
I20260809 04:55:04.416414 696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:55:04.416536 696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:55:04.416684 696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:04.418741 696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338f5d26ea14d58a526a8da5784779e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39179 } }
I20260809 04:55:04.419175 696 leader_election.cc:302] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7338f5d26ea14d58a526a8da5784779e; no voters:
I20260809 04:55:04.419695 696 leader_election.cc:288] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:55:04.419795 708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:04.420404 708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [term 1 LEADER]: Becoming Leader. State: Replica: 7338f5d26ea14d58a526a8da5784779e, State: Running, Role: LEADER
I20260809 04:55:04.420873 708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338f5d26ea14d58a526a8da5784779e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39179 } }
I20260809 04:55:04.421484 696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:04.423599 709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7338f5d26ea14d58a526a8da5784779e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338f5d26ea14d58a526a8da5784779e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39179 } } }
I20260809 04:55:04.423655 711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7338f5d26ea14d58a526a8da5784779e. Latest consensus state: current_term: 1 leader_uuid: "7338f5d26ea14d58a526a8da5784779e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338f5d26ea14d58a526a8da5784779e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39179 } } }
I20260809 04:55:04.424033 709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:04.424074 711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:04.425402 715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:04.429872 715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:55:04.437661 715 catalog_manager.cc:1347] Generated new cluster ID: c867805157a042b794b1fd5c0838c489
I20260809 04:55:04.437762 715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:04.453116 715 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:04.453589 715 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:04.465600 715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7338f5d26ea14d58a526a8da5784779e: Generated new TSK 0
I20260809 04:55:04.466538 715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:04.477070 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 583
W20260809 04:55:04.478408 683 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59933
W20260809 04:55:04.479107 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:55:04.479254 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 583
2026-08-09T04:55:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:55:04 dist-test-slave-4tdj krb5kdc[740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:55:04 dist-test-slave-4tdj krb5kdc[740](info): set up 2 sockets
Aug 09 04:55:04 dist-test-slave-4tdj krb5kdc[740](info): commencing operation
krb5kdc: starting...
W20260809 04:55:05.705024 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:55:05 dist-test-slave-4tdj krb5kdc[740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:55:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:55:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:55:05.954504 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:06.272030 761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:06.272280 761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:06.272405 761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:06.279273 761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:06.279371 761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:06.279438 761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:06.279529 761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:06.279590 761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:06.288800 761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:06.289945 761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:06.291237 761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:06.302196 772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:06.302280 767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:06.302349 768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:06.303288 761 server_base.cc:1034] running on GCE node
I20260809 04:55:06.304045 761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:06.305332 761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:06.306525 761 hybrid_clock.cc:648] HybridClock initialized: now 1786251306306472 us; error 47 us; skew 500 ppm
I20260809 04:55:06.306854 761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:06 dist-test-slave-4tdj krb5kdc[740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:06.309741 761 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:55:06.311141 761 webserver.cc:460] Webserver started at http://127.0.0.1:38849/ using document root <none> and password file <none>
I20260809 04:55:06.311764 761 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:06.311879 761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:06.312157 761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:06.315019 761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "4fdc79bda41f4d40aa415c7d1f577f5a"
format_stamp: "Formatted at 2026-08-09 04:55:06 on dist-test-slave-4tdj"
server_key: "7ab9d15db942f837dd21de5685543b4a"
server_key_iv: "c54ce5e4ec9db7f53438f648e73e11ad"
server_key_version: "encryptionkey@0"
I20260809 04:55:06.315793 761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "4fdc79bda41f4d40aa415c7d1f577f5a"
format_stamp: "Formatted at 2026-08-09 04:55:06 on dist-test-slave-4tdj"
server_key: "7ab9d15db942f837dd21de5685543b4a"
server_key_iv: "c54ce5e4ec9db7f53438f648e73e11ad"
server_key_version: "encryptionkey@0"
I20260809 04:55:06.320621 761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:55:06.323897 785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:06.325093 761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:55:06.325259 761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "4fdc79bda41f4d40aa415c7d1f577f5a"
format_stamp: "Formatted at 2026-08-09 04:55:06 on dist-test-slave-4tdj"
server_key: "7ab9d15db942f837dd21de5685543b4a"
server_key_iv: "c54ce5e4ec9db7f53438f648e73e11ad"
server_key_version: "encryptionkey@0"
I20260809 04:55:06.325457 761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:06.338923 761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:06.342777 761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:06.343019 761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:06.360204 761 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45755
I20260809 04:55:06.360235 854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45755 every 8 connection(s)
I20260809 04:55:06.361464 761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:06.364933 856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:06.366117 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 761
I20260809 04:55:06.366269 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:06.366724 32116 external_mini_cluster.cc:1428] Setting key 5093fb779368d21df70bf47caf7e1160
I20260809 04:55:06.378166 856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a: Bootstrap starting.
Aug 09 04:55:06 dist-test-slave-4tdj krb5kdc[740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 04:55:06.381819 856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:06.382921 856 log.cc:824] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:06.385716 856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a: No bootstrap required, opened a new log
I20260809 04:55:06.388847 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:06.368439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:34417 (local address 127.0.0.1:45755)
0809 04:55:06.369171 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:06.369183 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:06.369222 (+ 39us) server_negotiation.cc:407] Connection header received
0809 04:55:06.370850 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:06.370870 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:06.371296 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:06.371643 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:06.373295 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:06.374599 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:06.376348 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:06.376967 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:06.377083 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:06.381897 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:06.381906 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:06.381925 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:06.381971 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:06.384807 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:06.385654 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:06.385659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:06.385664 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:06.385851 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:06.386361 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:06.386364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:06.386366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:06.386852 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:06.387079 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:06.387586 (+ 507us) server_negotiation.cc:299] Negotiation successful
0809 04:55:06.387852 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":330,"threads_started":1}
I20260809 04:55:06.392336 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:06.394037 856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdc79bda41f4d40aa415c7d1f577f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } }
I20260809 04:55:06.394441 856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:55:06.394549 856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fdc79bda41f4d40aa415c7d1f577f5a, State: Initialized, Role: FOLLOWER
I20260809 04:55:06.395154 856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdc79bda41f4d40aa415c7d1f577f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } }
I20260809 04:55:06.395431 856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:55:06.395586 856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:55:06.395756 856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:06.398404 856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdc79bda41f4d40aa415c7d1f577f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } }
I20260809 04:55:06.398898 856 leader_election.cc:302] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fdc79bda41f4d40aa415c7d1f577f5a; no voters:
I20260809 04:55:06.399452 856 leader_election.cc:288] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:55:06.399583 875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:06.401696 856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:06.402565 875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [term 1 LEADER]: Becoming Leader. State: Replica: 4fdc79bda41f4d40aa415c7d1f577f5a, State: Running, Role: LEADER
I20260809 04:55:06.403261 875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdc79bda41f4d40aa415c7d1f577f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } }
I20260809 04:55:06.410732 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:06.397270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:38371 (local address 127.0.0.1:45755)
0809 04:55:06.397437 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:06.397442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:06.397912 (+ 470us) server_negotiation.cc:407] Connection header received
0809 04:55:06.398026 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:06.398030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:06.398103 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:06.398200 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:06.399614 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:06.400725 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:06.401954 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:06.402509 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:06.402674 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:06.405215 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:06.405230 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:06.405233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:06.405262 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:06.407845 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:06.408511 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:06.408516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:06.408519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:06.408651 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:06.409099 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:06.409102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:06.409104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:06.409369 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:06.409608 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:06.409976 (+ 368us) server_negotiation.cc:299] Negotiation successful
0809 04:55:06.410377 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:55:06.414031 876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fdc79bda41f4d40aa415c7d1f577f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdc79bda41f4d40aa415c7d1f577f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } } }
I20260809 04:55:06.414429 876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:06.420987 907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:06.420861 877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fdc79bda41f4d40aa415c7d1f577f5a. Latest consensus state: current_term: 1 leader_uuid: "4fdc79bda41f4d40aa415c7d1f577f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdc79bda41f4d40aa415c7d1f577f5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45755 } } }
I20260809 04:55:06.421257 877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:06.426065 907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:55:06.434561 907 catalog_manager.cc:1347] Generated new cluster ID: fd60919a7b6e4459b504724465e3ed3f
I20260809 04:55:06.434669 907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:06.456699 907 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:06.458237 907 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:06.466315 907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fdc79bda41f4d40aa415c7d1f577f5a: Generated new TSK 0
I20260809 04:55:06.467005 907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:06.472952 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 761
I20260809 04:55:06.474366 850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:55:06.560943 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 761
W20260809 04:55:06.584095 736 connection.cc:535] client connection to 127.0.0.1:45755 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45755): Connection reset by peer (error 104)
2026-08-09T04:55:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2082 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:55:06 dist-test-slave-4tdj krb5kdc[920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:55:06 dist-test-slave-4tdj krb5kdc[920](info): set up 2 sockets
Aug 09 04:55:06 dist-test-slave-4tdj krb5kdc[920](info): commencing operation
krb5kdc: starting...
W20260809 04:55:07.764061 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.004s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:55:07 dist-test-slave-4tdj krb5kdc[920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:55:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:55:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:55:08.024960 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:08.343448 941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:08.343708 941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:08.343804 941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:08.350711 941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:08.350807 941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:08.350858 941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:08.350915 941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:08.350957 941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:08.360255 941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:08.361408 941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:08.362654 941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:08.373502 950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:08.373509 954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:08.373492 951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:08.374696 941 server_base.cc:1034] running on GCE node
I20260809 04:55:08.375373 941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:08.376637 941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:08.377827 941 hybrid_clock.cc:648] HybridClock initialized: now 1786251308377774 us; error 43 us; skew 500 ppm
I20260809 04:55:08.378124 941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:08 dist-test-slave-4tdj krb5kdc[920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:08.380968 941 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:55:08.382390 941 webserver.cc:460] Webserver started at http://127.0.0.1:42713/ using document root <none> and password file <none>
I20260809 04:55:08.382972 941 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:08.383078 941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:08.383318 941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:08.386099 941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "a00d9518471546f2a45ebd8f64ad8a94"
format_stamp: "Formatted at 2026-08-09 04:55:08 on dist-test-slave-4tdj"
server_key: "046c18be5c909e8acb0f63f590cf3faf"
server_key_iv: "f65055dcc5be8c9f89733787ff9ae7b1"
server_key_version: "encryptionkey@0"
I20260809 04:55:08.386858 941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "a00d9518471546f2a45ebd8f64ad8a94"
format_stamp: "Formatted at 2026-08-09 04:55:08 on dist-test-slave-4tdj"
server_key: "046c18be5c909e8acb0f63f590cf3faf"
server_key_iv: "f65055dcc5be8c9f89733787ff9ae7b1"
server_key_version: "encryptionkey@0"
I20260809 04:55:08.391546 941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:55:08.394711 962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:08.395918 941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:55:08.396077 941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "a00d9518471546f2a45ebd8f64ad8a94"
format_stamp: "Formatted at 2026-08-09 04:55:08 on dist-test-slave-4tdj"
server_key: "046c18be5c909e8acb0f63f590cf3faf"
server_key_iv: "f65055dcc5be8c9f89733787ff9ae7b1"
server_key_version: "encryptionkey@0"
I20260809 04:55:08.396242 941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:08.425027 941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:08.428718 941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:08.428962 941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:08.445885 941 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45171
I20260809 04:55:08.445917 1048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45171 every 8 connection(s)
I20260809 04:55:08.447203 941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:08.450723 1050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:08.457402 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 941
I20260809 04:55:08.457609 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:08.458165 32116 external_mini_cluster.cc:1428] Setting key 2e46329476bab4a0e12549dfbae51585
I20260809 04:55:08.461546 1050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94: Bootstrap starting.
I20260809 04:55:08.464780 1050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:08.466032 1050 log.cc:824] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:08.469133 1050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94: No bootstrap required, opened a new log
I20260809 04:55:08.471371 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:08.459941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:36665 (local address 127.0.0.1:45171)
0809 04:55:08.460668 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:08.460678 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:08.460718 (+ 40us) server_negotiation.cc:407] Connection header received
0809 04:55:08.462388 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:08.462412 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:08.462774 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:08.463094 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:08.463661 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:08.464891 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:08.466823 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:08.467680 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:08.467801 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:08.469054 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:08.469061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:08.469073 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:08.469622 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:08.469694 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:08.469718 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:08.469831 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:08.470189 (+ 358us) server_negotiation.cc:299] Negotiation successful
0809 04:55:08.470394 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":326,"threads_started":1}
I20260809 04:55:08.474933 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:08.477588 1050 raft_consensus.cc:348] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00d9518471546f2a45ebd8f64ad8a94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } }
I20260809 04:55:08.478025 1050 raft_consensus.cc:374] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:55:08.478155 1050 raft_consensus.cc:729] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a00d9518471546f2a45ebd8f64ad8a94, State: Initialized, Role: FOLLOWER
I20260809 04:55:08.478863 1050 consensus_queue.cc:260] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00d9518471546f2a45ebd8f64ad8a94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } }
I20260809 04:55:08.479174 1050 raft_consensus.cc:388] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:55:08.479319 1050 raft_consensus.cc:482] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:55:08.479499 1050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:08.482329 1050 raft_consensus.cc:504] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00d9518471546f2a45ebd8f64ad8a94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } }
I20260809 04:55:08.482831 1050 leader_election.cc:302] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a00d9518471546f2a45ebd8f64ad8a94; no voters:
I20260809 04:55:08.483389 1050 leader_election.cc:288] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:55:08.483516 1071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:08.484135 1071 raft_consensus.cc:686] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [term 1 LEADER]: Becoming Leader. State: Replica: a00d9518471546f2a45ebd8f64ad8a94, State: Running, Role: LEADER
I20260809 04:55:08.484742 1071 consensus_queue.cc:237] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00d9518471546f2a45ebd8f64ad8a94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } }
I20260809 04:55:08.486527 1071 sys_catalog.cc:455] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a00d9518471546f2a45ebd8f64ad8a94. Latest consensus state: current_term: 1 leader_uuid: "a00d9518471546f2a45ebd8f64ad8a94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00d9518471546f2a45ebd8f64ad8a94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } } }
I20260809 04:55:08.486846 1071 sys_catalog.cc:458] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:08.487403 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:08.480131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:58521 (local address 127.0.0.1:45171)
0809 04:55:08.480320 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:08.480326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:08.480707 (+ 381us) server_negotiation.cc:407] Connection header received
0809 04:55:08.480835 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:08.480844 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:08.480920 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:08.481019 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:08.481447 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:08.482506 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:08.484214 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:08.484893 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:08.485015 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:08.486405 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:08.486416 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:08.486421 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:08.486444 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:08.486525 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:08.486547 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:08.486646 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:08.486972 (+ 326us) server_negotiation.cc:299] Negotiation successful
0809 04:55:08.487031 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:55:08.487283 1073 sys_catalog.cc:455] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a00d9518471546f2a45ebd8f64ad8a94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00d9518471546f2a45ebd8f64ad8a94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45171 } } }
I20260809 04:55:08.487627 1073 sys_catalog.cc:458] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:08.491504 1050 sys_catalog.cc:564] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:08.493525 1076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:08.498533 1076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:55:08.508026 1076 catalog_manager.cc:1347] Generated new cluster ID: ff7ece644fe346689fe5523c0930f75f
I20260809 04:55:08.508143 1076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:08.538923 1076 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:08.540228 1076 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:08.553654 1076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a00d9518471546f2a45ebd8f64ad8a94: Generated new TSK 0
I20260809 04:55:08.554518 1076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:08.569445 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 941
W20260809 04:55:08.570538 1044 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.56:36665
W20260809 04:55:08.571288 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 941: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:55:08.571429 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 941
W20260809 04:55:08.597688 917 connection.cc:535] client connection to 127.0.0.1:45171 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45171): Connection reset by peer (error 104)
2026-08-09T04:55:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2013 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:55:08 dist-test-slave-4tdj krb5kdc[1104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:55:08 dist-test-slave-4tdj krb5kdc[1104](info): set up 2 sockets
Aug 09 04:55:08 dist-test-slave-4tdj krb5kdc[1104](info): commencing operation
krb5kdc: starting...
W20260809 04:55:09.793671 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:55:09 dist-test-slave-4tdj krb5kdc[1104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:55:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:55:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:55:10.058435 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37159
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:10.378625 1130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:10.378876 1130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:10.379006 1130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:10.385784 1130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:10.385885 1130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:10.385958 1130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:10.386039 1130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:10.386098 1130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:10.395392 1130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:10.396697 1130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:10.397931 1130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:10.408825 1139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:10.408915 1142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:10.408861 1140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:10.409617 1130 server_base.cc:1034] running on GCE node
I20260809 04:55:10.410323 1130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:10.411574 1130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:10.412771 1130 hybrid_clock.cc:648] HybridClock initialized: now 1786251310412719 us; error 46 us; skew 500 ppm
I20260809 04:55:10.413079 1130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:10 dist-test-slave-4tdj krb5kdc[1104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:10.420396 1130 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:55:10.421789 1130 webserver.cc:460] Webserver started at http://127.0.0.1:39297/ using document root <none> and password file <none>
I20260809 04:55:10.422433 1130 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:10.422554 1130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:10.422840 1130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:10.425695 1130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "4e26c667bbcb4458a9604e32bd9f11e5"
format_stamp: "Formatted at 2026-08-09 04:55:10 on dist-test-slave-4tdj"
server_key: "3002e4a6d7c6f3e0ec2eb10373c425a4"
server_key_iv: "d5a536a02b6fe1966bdbbdcb0cc812d1"
server_key_version: "encryptionkey@0"
I20260809 04:55:10.426461 1130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "4e26c667bbcb4458a9604e32bd9f11e5"
format_stamp: "Formatted at 2026-08-09 04:55:10 on dist-test-slave-4tdj"
server_key: "3002e4a6d7c6f3e0ec2eb10373c425a4"
server_key_iv: "d5a536a02b6fe1966bdbbdcb0cc812d1"
server_key_version: "encryptionkey@0"
I20260809 04:55:10.431190 1130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 04:55:10.434351 1152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:10.435508 1130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 04:55:10.435672 1130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "4e26c667bbcb4458a9604e32bd9f11e5"
format_stamp: "Formatted at 2026-08-09 04:55:10 on dist-test-slave-4tdj"
server_key: "3002e4a6d7c6f3e0ec2eb10373c425a4"
server_key_iv: "d5a536a02b6fe1966bdbbdcb0cc812d1"
server_key_version: "encryptionkey@0"
I20260809 04:55:10.435839 1130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:10.441128 1130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:10.444669 1130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:10.444917 1130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:10.461503 1130 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37159
I20260809 04:55:10.461557 1238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37159 every 8 connection(s)
I20260809 04:55:10.462757 1130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:10.466141 1239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:10.471176 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 1130
I20260809 04:55:10.471342 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:10.471877 32116 external_mini_cluster.cc:1428] Setting key 1a28ce8cfdecd9cac6049b2959ee0f8e
I20260809 04:55:10.476852 1239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5: Bootstrap starting.
I20260809 04:55:10.479094 1242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:10.473581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:43407 (local address 127.0.0.1:37159)
0809 04:55:10.474354 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:10.474365 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:10.474408 (+ 43us) server_negotiation.cc:407] Connection header received
0809 04:55:10.476026 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:10.476048 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:10.476360 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:10.476687 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:10.477593 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:10.477599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:10.477612 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:10.477626 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:10.477681 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:10.477700 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:10.477894 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:10.478036 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 04:55:10.478226 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":314,"threads_started":1}
I20260809 04:55:10.480249 1239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:10.481350 1239 log.cc:824] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:10.482615 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:10.484355 1239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5: No bootstrap required, opened a new log
I20260809 04:55:10.489866 1242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:10.487705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:37975 (local address 127.0.0.1:37159)
0809 04:55:10.487879 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:10.487884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:10.488201 (+ 317us) server_negotiation.cc:407] Connection header received
0809 04:55:10.488350 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:10.488353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:10.488398 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:10.488471 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:10.489306 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:10.489310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:10.489313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:10.489326 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:10.489359 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:10.489372 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:10.489456 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:10.489587 (+ 131us) server_negotiation.cc:299] Negotiation successful
0809 04:55:10.489627 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:55:10.493057 1239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e26c667bbcb4458a9604e32bd9f11e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37159 } }
I20260809 04:55:10.493410 1239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:55:10.493511 1239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e26c667bbcb4458a9604e32bd9f11e5, State: Initialized, Role: FOLLOWER
I20260809 04:55:10.494005 1239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e26c667bbcb4458a9604e32bd9f11e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37159 } }
I20260809 04:55:10.494235 1239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:55:10.494434 1239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:55:10.494635 1239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:10.496944 1239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e26c667bbcb4458a9604e32bd9f11e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37159 } }
I20260809 04:55:10.497404 1239 leader_election.cc:302] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e26c667bbcb4458a9604e32bd9f11e5; no voters:
I20260809 04:55:10.498031 1239 leader_election.cc:288] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:55:10.498128 1251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:10.499123 1251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [term 1 LEADER]: Becoming Leader. State: Replica: 4e26c667bbcb4458a9604e32bd9f11e5, State: Running, Role: LEADER
I20260809 04:55:10.499603 1251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e26c667bbcb4458a9604e32bd9f11e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37159 } }
I20260809 04:55:10.499907 1239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:10.502099 1253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e26c667bbcb4458a9604e32bd9f11e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e26c667bbcb4458a9604e32bd9f11e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37159 } } }
I20260809 04:55:10.502139 1255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e26c667bbcb4458a9604e32bd9f11e5. Latest consensus state: current_term: 1 leader_uuid: "4e26c667bbcb4458a9604e32bd9f11e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e26c667bbcb4458a9604e32bd9f11e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37159 } } }
I20260809 04:55:10.502539 1253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:10.502569 1255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:10.503400 1262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:10.509439 1262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:55:10.520352 1262 catalog_manager.cc:1347] Generated new cluster ID: 5eafce23decc4f3ba99d8802a46185f6
I20260809 04:55:10.520457 1262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:10.543118 1262 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:10.543534 1262 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:10.553308 1262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e26c667bbcb4458a9604e32bd9f11e5: Generated new TSK 0
I20260809 04:55:10.554087 1262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:10.570766 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 1130
W20260809 04:55:10.571807 1232 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.56:43407
W20260809 04:55:10.572504 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 1130: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:55:10.572647 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 1130
2026-08-09T04:55:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1996 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:55:10 dist-test-slave-4tdj krb5kdc[1288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:55:10 dist-test-slave-4tdj krb5kdc[1288](info): set up 2 sockets
Aug 09 04:55:10 dist-test-slave-4tdj krb5kdc[1288](info): commencing operation
krb5kdc: starting...
W20260809 04:55:11.779911 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.006s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:55:11 dist-test-slave-4tdj krb5kdc[1288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:55:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:55:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:55:12.043361 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34609
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:12.364363 1321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:12.364651 1321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:12.364751 1321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:12.371862 1321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:12.371973 1321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:12.372026 1321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:12.372085 1321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:12.372128 1321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:12.381573 1321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34609
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:12.382761 1321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:12.383975 1321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:12.394722 1332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:12.394762 1335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:12.394807 1333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:12.395563 1321 server_base.cc:1034] running on GCE node
I20260809 04:55:12.396242 1321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:12.397487 1321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:12.398679 1321 hybrid_clock.cc:648] HybridClock initialized: now 1786251312398631 us; error 41 us; skew 500 ppm
I20260809 04:55:12.398991 1321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:12 dist-test-slave-4tdj krb5kdc[1288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:12.401878 1321 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:55:12.403194 1321 webserver.cc:460] Webserver started at http://127.0.0.1:38493/ using document root <none> and password file <none>
I20260809 04:55:12.403796 1321 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:12.403913 1321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:12.404194 1321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:12.407053 1321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "9ec064c53c3845ebb4d30304a729359a"
format_stamp: "Formatted at 2026-08-09 04:55:12 on dist-test-slave-4tdj"
server_key: "21e63522f11c4b9957c919cdc736a46a"
server_key_iv: "2fe6938397d8b99ad3acbfd747aaa52b"
server_key_version: "encryptionkey@0"
I20260809 04:55:12.407778 1321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "9ec064c53c3845ebb4d30304a729359a"
format_stamp: "Formatted at 2026-08-09 04:55:12 on dist-test-slave-4tdj"
server_key: "21e63522f11c4b9957c919cdc736a46a"
server_key_iv: "2fe6938397d8b99ad3acbfd747aaa52b"
server_key_version: "encryptionkey@0"
I20260809 04:55:12.412581 1321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 04:55:12.415820 1346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:12.416999 1321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:55:12.417166 1321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "9ec064c53c3845ebb4d30304a729359a"
format_stamp: "Formatted at 2026-08-09 04:55:12 on dist-test-slave-4tdj"
server_key: "21e63522f11c4b9957c919cdc736a46a"
server_key_iv: "2fe6938397d8b99ad3acbfd747aaa52b"
server_key_version: "encryptionkey@0"
I20260809 04:55:12.417335 1321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:12.432164 1321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:12.435731 1321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:12.435979 1321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:12.452754 1321 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34609
I20260809 04:55:12.452790 1428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34609 every 8 connection(s)
I20260809 04:55:12.453999 1321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:12.457504 1430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:12.457784 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 1321
I20260809 04:55:12.457938 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:12.458434 32116 external_mini_cluster.cc:1428] Setting key 0bcc1f08db3661b37de333e7ed1c8e40
I20260809 04:55:12.471361 1430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a: Bootstrap starting.
Aug 09 04:55:12 dist-test-slave-4tdj krb5kdc[1288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 04:55:12.474574 1430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:12.475911 1430 log.cc:824] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:12.479329 1430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a: No bootstrap required, opened a new log
I20260809 04:55:12.483197 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:12.460133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:60931 (local address 127.0.0.1:34609)
0809 04:55:12.460920 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:12.460934 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:12.460994 (+ 60us) server_negotiation.cc:407] Connection header received
0809 04:55:12.463745 (+ 2751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:12.463774 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:12.464273 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:12.464695 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:12.465949 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:12.467891 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:12.469209 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:12.470223 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:12.470422 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:12.474824 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:12.474836 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:12.474853 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:12.474906 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:12.478710 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:12.479486 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:12.479494 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:12.479502 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:12.479724 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:12.480138 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:12.480142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:12.480145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:12.480815 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:12.481134 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:12.481513 (+ 379us) server_negotiation.cc:299] Negotiation successful
0809 04:55:12.481859 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":305,"threads_started":1}
I20260809 04:55:12.486924 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:12.488480 1430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec064c53c3845ebb4d30304a729359a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34609 } }
I20260809 04:55:12.488864 1430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:55:12.488955 1430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec064c53c3845ebb4d30304a729359a, State: Initialized, Role: FOLLOWER
I20260809 04:55:12.489567 1430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec064c53c3845ebb4d30304a729359a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34609 } }
I20260809 04:55:12.489848 1430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:55:12.489996 1430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:55:12.490180 1430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:12.493153 1430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec064c53c3845ebb4d30304a729359a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34609 } }
I20260809 04:55:12.493711 1430 leader_election.cc:302] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ec064c53c3845ebb4d30304a729359a; no voters:
I20260809 04:55:12.494339 1430 leader_election.cc:288] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:55:12.494432 1452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:12.496347 1430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:12.498591 1452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [term 1 LEADER]: Becoming Leader. State: Replica: 9ec064c53c3845ebb4d30304a729359a, State: Running, Role: LEADER
I20260809 04:55:12.499101 1452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec064c53c3845ebb4d30304a729359a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34609 } }
I20260809 04:55:12.501541 1456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ec064c53c3845ebb4d30304a729359a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec064c53c3845ebb4d30304a729359a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34609 } } }
I20260809 04:55:12.501531 1457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ec064c53c3845ebb4d30304a729359a. Latest consensus state: current_term: 1 leader_uuid: "9ec064c53c3845ebb4d30304a729359a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec064c53c3845ebb4d30304a729359a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34609 } } }
I20260809 04:55:12.501965 1457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:12.501971 1456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:12.509850 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:12.491991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:37653 (local address 127.0.0.1:34609)
0809 04:55:12.492197 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:12.492203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:12.492557 (+ 354us) server_negotiation.cc:407] Connection header received
0809 04:55:12.492704 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:12.492707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:12.492761 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:12.492834 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:12.494700 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:12.495863 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:12.497753 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:12.498470 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:12.498628 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:12.501861 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:12.501869 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:12.501875 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:12.501920 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:12.505938 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:12.506972 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:12.506978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:12.506982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:12.507131 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:12.507701 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:12.507707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:12.507710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:12.508124 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:12.508357 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:12.509066 (+ 709us) server_negotiation.cc:299] Negotiation successful
0809 04:55:12.509461 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 04:55:12.516590 1462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:12.521514 1462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:55:12.530246 1462 catalog_manager.cc:1347] Generated new cluster ID: 1bb5ebe5cbab4bb28784e15bc803199a
I20260809 04:55:12.530369 1462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:12.541875 1462 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:12.543102 1462 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:12.556345 1462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ec064c53c3845ebb4d30304a729359a: Generated new TSK 0
I20260809 04:55:12.557076 1462 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:12.568414 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 1321
I20260809 04:55:12.569859 1423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:55:12.658164 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 1321
W20260809 04:55:12.678516 1283 connection.cc:535] client connection to 127.0.0.1:34609 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34609): Connection reset by peer (error 104)
2026-08-09T04:55:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2083 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:55:12 dist-test-slave-4tdj krb5kdc[1489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:55:12 dist-test-slave-4tdj krb5kdc[1489](info): set up 2 sockets
Aug 09 04:55:12 dist-test-slave-4tdj krb5kdc[1489](info): commencing operation
krb5kdc: starting...
W20260809 04:55:13.859665 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:55:14 dist-test-slave-4tdj krb5kdc[1489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:55:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:55:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:55:14.114502 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:14.440886 1515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:14.441151 1515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:14.441280 1515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:14.448207 1515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:14.448315 1515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:14.448388 1515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:14.448474 1515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:14.448529 1515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:14.457898 1515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35585
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:14.459096 1515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:14.460314 1515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:14.471040 1528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:14.471107 1530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:14.471081 1532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:14.472146 1515 server_base.cc:1034] running on GCE node
I20260809 04:55:14.472872 1515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:14.474062 1515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:14.475247 1515 hybrid_clock.cc:648] HybridClock initialized: now 1786251314475192 us; error 50 us; skew 500 ppm
I20260809 04:55:14.475556 1515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:14 dist-test-slave-4tdj krb5kdc[1489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:14.478480 1515 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:55:14.479830 1515 webserver.cc:460] Webserver started at http://127.0.0.1:37091/ using document root <none> and password file <none>
I20260809 04:55:14.480438 1515 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:14.480544 1515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:14.480825 1515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:14.483618 1515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "7cc815ebb19d4936ad95be49933b6fdc"
format_stamp: "Formatted at 2026-08-09 04:55:14 on dist-test-slave-4tdj"
server_key: "74649837d40275a855c8a5188fbf3433"
server_key_iv: "e92d5a3e32652a18aafb790a3af5bfac"
server_key_version: "encryptionkey@0"
I20260809 04:55:14.484311 1515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "7cc815ebb19d4936ad95be49933b6fdc"
format_stamp: "Formatted at 2026-08-09 04:55:14 on dist-test-slave-4tdj"
server_key: "74649837d40275a855c8a5188fbf3433"
server_key_iv: "e92d5a3e32652a18aafb790a3af5bfac"
server_key_version: "encryptionkey@0"
I20260809 04:55:14.489041 1515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 04:55:14.492221 1541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:14.493404 1515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:55:14.493566 1515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "7cc815ebb19d4936ad95be49933b6fdc"
format_stamp: "Formatted at 2026-08-09 04:55:14 on dist-test-slave-4tdj"
server_key: "74649837d40275a855c8a5188fbf3433"
server_key_iv: "e92d5a3e32652a18aafb790a3af5bfac"
server_key_version: "encryptionkey@0"
I20260809 04:55:14.493746 1515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:14.512468 1515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:14.516182 1515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:14.516414 1515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:14.533497 1515 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35585
I20260809 04:55:14.533540 1627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35585 every 8 connection(s)
I20260809 04:55:14.534763 1515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:14.537267 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 1515
I20260809 04:55:14.537422 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:14.537845 32116 external_mini_cluster.cc:1428] Setting key 5e4eb21dfe285f827fe28f32a5951e19
I20260809 04:55:14.538132 1629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:14.550781 1635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:14.539441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:53479 (local address 127.0.0.1:35585)
0809 04:55:14.540217 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:14.540227 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:14.540266 (+ 39us) server_negotiation.cc:407] Connection header received
0809 04:55:14.542377 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:14.542398 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:14.542753 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:14.543064 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:14.543616 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:14.544906 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:14.546678 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:14.547313 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:14.547446 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:14.548660 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:14.548671 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:14.548682 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:14.549206 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:14.549285 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:14.549310 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:14.549419 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:14.549669 (+ 250us) server_negotiation.cc:299] Negotiation successful
0809 04:55:14.549831 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":313,"threads_started":1}
I20260809 04:55:14.550911 1629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc: Bootstrap starting.
I20260809 04:55:14.553891 1629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:14.554113 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:14.554908 1629 log.cc:824] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:14.557489 1629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc: No bootstrap required, opened a new log
I20260809 04:55:14.563974 1635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:14.558330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:47951 (local address 127.0.0.1:35585)
0809 04:55:14.558484 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:14.558489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:14.558849 (+ 360us) server_negotiation.cc:407] Connection header received
0809 04:55:14.558990 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:14.558995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:14.559057 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:14.559157 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:14.559626 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:14.560388 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:14.561718 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:14.562177 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:14.562311 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:14.563308 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:14.563313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:14.563316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:14.563332 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:14.563384 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:14.563398 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:14.563495 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:14.563698 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 04:55:14.563751 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 04:55:14.565279 1629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc815ebb19d4936ad95be49933b6fdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35585 } }
I20260809 04:55:14.565591 1629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:55:14.565680 1629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cc815ebb19d4936ad95be49933b6fdc, State: Initialized, Role: FOLLOWER
I20260809 04:55:14.566164 1629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc815ebb19d4936ad95be49933b6fdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35585 } }
I20260809 04:55:14.566396 1629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:55:14.566514 1629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:55:14.566664 1629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:14.568614 1629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc815ebb19d4936ad95be49933b6fdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35585 } }
I20260809 04:55:14.569051 1629 leader_election.cc:302] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cc815ebb19d4936ad95be49933b6fdc; no voters:
I20260809 04:55:14.569545 1629 leader_election.cc:288] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:55:14.569717 1647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:14.570449 1647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [term 1 LEADER]: Becoming Leader. State: Replica: 7cc815ebb19d4936ad95be49933b6fdc, State: Running, Role: LEADER
I20260809 04:55:14.571043 1647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc815ebb19d4936ad95be49933b6fdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35585 } }
I20260809 04:55:14.571447 1629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:14.573669 1649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cc815ebb19d4936ad95be49933b6fdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc815ebb19d4936ad95be49933b6fdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35585 } } }
I20260809 04:55:14.573694 1648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cc815ebb19d4936ad95be49933b6fdc. Latest consensus state: current_term: 1 leader_uuid: "7cc815ebb19d4936ad95be49933b6fdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cc815ebb19d4936ad95be49933b6fdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35585 } } }
I20260809 04:55:14.574153 1648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:14.579097 1649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:14.579309 1654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:14.584647 1654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:55:14.593732 1654 catalog_manager.cc:1347] Generated new cluster ID: 418f0189eebc4479b54359b21e7e8b6f
I20260809 04:55:14.593837 1654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:14.608431 1654 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:14.609927 1654 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:14.625002 1654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cc815ebb19d4936ad95be49933b6fdc: Generated new TSK 0
I20260809 04:55:14.625654 1654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:14.631803 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 1515
W20260809 04:55:14.632856 1620 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.56:53479
W20260809 04:55:14.633596 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 1515: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:55:14.633736 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 1515
W20260809 04:55:14.659127 1487 connection.cc:535] client connection to 127.0.0.1:35585 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35585): Connection reset by peer (error 104)
2026-08-09T04:55:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1981 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:55:14 dist-test-slave-4tdj krb5kdc[1684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:55:14 dist-test-slave-4tdj krb5kdc[1684](info): set up 2 sockets
Aug 09 04:55:14 dist-test-slave-4tdj krb5kdc[1684](info): commencing operation
krb5kdc: starting...
W20260809 04:55:15.869468 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:55:16 dist-test-slave-4tdj krb5kdc[1684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:55:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:55:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 04:55:16.119467 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:16.439200 1701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:16.439455 1701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:16.439596 1701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:16.446684 1701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:16.446781 1701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:16.446834 1701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:16.446919 1701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:16.446972 1701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:16.456413 1701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:16.457583 1701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:16.458813 1701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:16.469635 1709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:16.469746 1712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:16.469818 1715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:16.470736 1701 server_base.cc:1034] running on GCE node
I20260809 04:55:16.471443 1701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:16.472707 1701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:16.473914 1701 hybrid_clock.cc:648] HybridClock initialized: now 1786251316473874 us; error 26 us; skew 500 ppm
I20260809 04:55:16.474221 1701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:16 dist-test-slave-4tdj krb5kdc[1684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:16.480006 1701 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 04:55:16.481401 1701 webserver.cc:460] Webserver started at http://127.0.0.1:39283/ using document root <none> and password file <none>
I20260809 04:55:16.482029 1701 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:16.482146 1701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:16.482462 1701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:16.485301 1701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "44f671e23a50453aba4cdbc8df451ca7"
format_stamp: "Formatted at 2026-08-09 04:55:16 on dist-test-slave-4tdj"
server_key: "168a47524d551271de5dbbcadc705b01"
server_key_iv: "0b36ab6a7cc39cbb8076332683ced21a"
server_key_version: "encryptionkey@0"
I20260809 04:55:16.486016 1701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "44f671e23a50453aba4cdbc8df451ca7"
format_stamp: "Formatted at 2026-08-09 04:55:16 on dist-test-slave-4tdj"
server_key: "168a47524d551271de5dbbcadc705b01"
server_key_iv: "0b36ab6a7cc39cbb8076332683ced21a"
server_key_version: "encryptionkey@0"
I20260809 04:55:16.490813 1701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 04:55:16.493927 1727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:16.495112 1701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:55:16.495272 1701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "44f671e23a50453aba4cdbc8df451ca7"
format_stamp: "Formatted at 2026-08-09 04:55:16 on dist-test-slave-4tdj"
server_key: "168a47524d551271de5dbbcadc705b01"
server_key_iv: "0b36ab6a7cc39cbb8076332683ced21a"
server_key_version: "encryptionkey@0"
I20260809 04:55:16.495446 1701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:16.500747 1701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:16.504225 1701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:16.504482 1701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:16.521258 1701 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37071
I20260809 04:55:16.521319 1807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37071 every 8 connection(s)
I20260809 04:55:16.522539 1701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:16.525980 1808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:16.531323 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 1701
I20260809 04:55:16.531507 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:16.532039 32116 external_mini_cluster.cc:1428] Setting key 3ca06d78677f385bf47791e0f65a712b
I20260809 04:55:16.536559 1808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7: Bootstrap starting.
I20260809 04:55:16.539674 1812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:16.533854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:52003 (local address 127.0.0.1:37071)
0809 04:55:16.534723 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:16.534735 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:16.534784 (+ 49us) server_negotiation.cc:407] Connection header received
0809 04:55:16.536486 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:16.536510 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:16.536849 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:16.537200 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:16.538220 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:16.538227 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:16.538239 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:16.538255 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:16.538354 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:16.538388 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:16.538562 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:16.538658 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 04:55:16.538817 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":385,"threads_started":1}
I20260809 04:55:16.539958 1808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:16.541222 1808 log.cc:824] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:16.543309 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:16.544164 1808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7: No bootstrap required, opened a new log
I20260809 04:55:16.550554 1812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:16.548268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:47695 (local address 127.0.0.1:37071)
0809 04:55:16.548434 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:16.548439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:16.548717 (+ 278us) server_negotiation.cc:407] Connection header received
0809 04:55:16.548871 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:16.548875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:16.548921 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:16.549003 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:16.549837 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:16.549841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:16.549844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:16.549858 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:16.549896 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:16.549910 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:16.550007 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:16.550156 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 04:55:16.550213 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 04:55:16.553356 1808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f671e23a50453aba4cdbc8df451ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } }
I20260809 04:55:16.553699 1808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:55:16.553798 1808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f671e23a50453aba4cdbc8df451ca7, State: Initialized, Role: FOLLOWER
I20260809 04:55:16.554342 1808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f671e23a50453aba4cdbc8df451ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } }
I20260809 04:55:16.554586 1808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:55:16.554713 1808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:55:16.554869 1808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:16.557224 1808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f671e23a50453aba4cdbc8df451ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } }
I20260809 04:55:16.557670 1808 leader_election.cc:302] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44f671e23a50453aba4cdbc8df451ca7; no voters:
I20260809 04:55:16.558176 1808 leader_election.cc:288] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:55:16.558280 1821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:16.558945 1821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [term 1 LEADER]: Becoming Leader. State: Replica: 44f671e23a50453aba4cdbc8df451ca7, State: Running, Role: LEADER
I20260809 04:55:16.559469 1821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f671e23a50453aba4cdbc8df451ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } }
I20260809 04:55:16.560074 1808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:16.562708 1825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f671e23a50453aba4cdbc8df451ca7. Latest consensus state: current_term: 1 leader_uuid: "44f671e23a50453aba4cdbc8df451ca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f671e23a50453aba4cdbc8df451ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } } }
I20260809 04:55:16.563153 1825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:16.563336 1822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44f671e23a50453aba4cdbc8df451ca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f671e23a50453aba4cdbc8df451ca7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } } }
I20260809 04:55:16.563663 1822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:16.567215 1838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:16.571817 1838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:55:16.579794 1838 catalog_manager.cc:1347] Generated new cluster ID: 7542673da4924ec191eff7317e9ae62a
I20260809 04:55:16.579931 1838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:16.595450 1838 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:16.595803 1838 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:16.601559 1838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44f671e23a50453aba4cdbc8df451ca7: Generated new TSK 0
I20260809 04:55:16.602447 1838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:16.612875 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 1701
W20260809 04:55:16.614045 1801 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.56:52003
W20260809 04:55:16.614681 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 1701: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:55:16.614811 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 1701
2026-08-09T04:55:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1978 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:55:16 dist-test-slave-4tdj krb5kdc[1853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:55:16 dist-test-slave-4tdj krb5kdc[1853](info): set up 2 sockets
Aug 09 04:55:16 dist-test-slave-4tdj krb5kdc[1853](info): commencing operation
krb5kdc: starting...
W20260809 04:55:17.831243 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.003s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:55:17 dist-test-slave-4tdj krb5kdc[1853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:55:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:55:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:55:18.074224 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:38549
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:38549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:18.388012 1875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:18.388262 1875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:18.388377 1875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:18.395257 1875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:18.395354 1875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:18.395428 1875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:18.395499 1875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:18.395562 1875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:18.404765 1875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:38549
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:38549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:18.405907 1875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:18.407150 1875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:18.417846 1883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:18.418282 1890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:18.417932 1884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:18.419098 1875 server_base.cc:1034] running on GCE node
I20260809 04:55:18.419790 1875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:18.421133 1875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:18.422356 1875 hybrid_clock.cc:648] HybridClock initialized: now 1786251318422311 us; error 37 us; skew 500 ppm
I20260809 04:55:18.422652 1875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:18 dist-test-slave-4tdj krb5kdc[1853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251318, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:18.425488 1875 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:55:18.426921 1875 webserver.cc:460] Webserver started at http://127.31.93.62:38779/ using document root <none> and password file <none>
I20260809 04:55:18.427512 1875 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:18.427623 1875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:18.427861 1875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:18.430670 1875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "92f59f57b984458d9273909ce6441e41"
format_stamp: "Formatted at 2026-08-09 04:55:18 on dist-test-slave-4tdj"
server_key: "fd5537ddd3402c436f52588c734fe920"
server_key_iv: "3ab47f9f1c4378871508309de749be6a"
server_key_version: "encryptionkey@0"
I20260809 04:55:18.431345 1875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "92f59f57b984458d9273909ce6441e41"
format_stamp: "Formatted at 2026-08-09 04:55:18 on dist-test-slave-4tdj"
server_key: "fd5537ddd3402c436f52588c734fe920"
server_key_iv: "3ab47f9f1c4378871508309de749be6a"
server_key_version: "encryptionkey@0"
I20260809 04:55:18.436165 1875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 04:55:18.439374 1903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:18.440586 1875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 04:55:18.440737 1875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "92f59f57b984458d9273909ce6441e41"
format_stamp: "Formatted at 2026-08-09 04:55:18 on dist-test-slave-4tdj"
server_key: "fd5537ddd3402c436f52588c734fe920"
server_key_iv: "3ab47f9f1c4378871508309de749be6a"
server_key_version: "encryptionkey@0"
I20260809 04:55:18.440917 1875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:18.452507 1875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:18.456048 1875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:18.456292 1875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:18.473287 1875 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:38549
I20260809 04:55:18.473325 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:38549 every 8 connection(s)
I20260809 04:55:18.474630 1875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:18.478031 1991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:18.483842 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 1875
I20260809 04:55:18.483997 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:18.484421 32116 external_mini_cluster.cc:1428] Setting key d77f1df7f96a0669457872a65965c30a
I20260809 04:55:18.488803 1991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41: Bootstrap starting.
I20260809 04:55:18.492435 1991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:18.493605 1991 log.cc:824] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:18.496608 1991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41: No bootstrap required, opened a new log
Aug 09 04:55:18 dist-test-slave-4tdj krb5kdc[1853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:55:18.505502 1991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f59f57b984458d9273909ce6441e41" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 38549 } }
I20260809 04:55:18.505965 1991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:55:18.506071 1991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f59f57b984458d9273909ce6441e41, State: Initialized, Role: FOLLOWER
I20260809 04:55:18.506752 1991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f59f57b984458d9273909ce6441e41" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 38549 } }
I20260809 04:55:18.507037 1991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:55:18.507211 1991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:55:18.507398 1991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:18.508162 1995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:18.486101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39613 (local address 127.31.93.62:38549)
0809 04:55:18.486916 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:18.486929 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:18.486976 (+ 47us) server_negotiation.cc:407] Connection header received
0809 04:55:18.489225 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:18.489252 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:18.489702 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:18.490105 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:18.491331 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:18.493234 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:18.494779 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:18.495758 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:18.495930 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:18.499705 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:18.499714 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:18.499731 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:18.499777 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:18.503585 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:18.504325 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:18.504333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:18.504341 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:18.504547 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:18.504935 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:18.504940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:18.504943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:18.505631 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:18.505972 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:18.506416 (+ 444us) server_negotiation.cc:299] Negotiation successful
0809 04:55:18.506753 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":356,"threads_started":1}
I20260809 04:55:18.509902 1991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f59f57b984458d9273909ce6441e41" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 38549 } }
I20260809 04:55:18.510408 1991 leader_election.cc:302] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92f59f57b984458d9273909ce6441e41; no voters:
I20260809 04:55:18.511016 1991 leader_election.cc:288] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:55:18.511157 1998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:18.511771 1998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [term 1 LEADER]: Becoming Leader. State: Replica: 92f59f57b984458d9273909ce6441e41, State: Running, Role: LEADER
I20260809 04:55:18.511919 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:18.512362 1998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f59f57b984458d9273909ce6441e41" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 38549 } }
I20260809 04:55:18.514050 1998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92f59f57b984458d9273909ce6441e41. Latest consensus state: current_term: 1 leader_uuid: "92f59f57b984458d9273909ce6441e41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f59f57b984458d9273909ce6441e41" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 38549 } } }
I20260809 04:55:18.515210 2001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92f59f57b984458d9273909ce6441e41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f59f57b984458d9273909ce6441e41" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 38549 } } }
I20260809 04:55:18.515523 2001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:18.515862 1998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:18.518833 1991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:18.519081 2005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:18.524591 2005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:55:18.543641 1995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:18.520002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47843 (local address 127.31.93.62:38549)
0809 04:55:18.520179 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:18.520185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:18.526722 (+ 6537us) server_negotiation.cc:407] Connection header received
0809 04:55:18.526981 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:18.526987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:18.527073 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:18.527147 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:18.529181 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:18.530378 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:18.532183 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:18.532843 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:18.532937 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:18.536297 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:18.536304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:18.536309 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:18.536351 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:18.539975 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:18.540985 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:18.540991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:18.540994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:18.541146 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:18.541756 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:18.541761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:18.541764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:18.542164 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:18.542420 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:18.543101 (+ 681us) server_negotiation.cc:299] Negotiation successful
0809 04:55:18.543280 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 04:55:18.547025 2005 catalog_manager.cc:1347] Generated new cluster ID: 5431330b47114564933370b32bc83459
I20260809 04:55:18.547142 2005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:18.580328 2005 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:18.581638 2005 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:18.595230 2005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92f59f57b984458d9273909ce6441e41: Generated new TSK 0
I20260809 04:55:18.596007 2005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:18.611316 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 1875
I20260809 04:55:18.612357 1985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:55:18.704295 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 1875
W20260809 04:55:18.726871 1850 connection.cc:535] client connection to 127.31.93.62:38549 recv error: Network error: failed to read from TLS socket (remote: 127.31.93.62:38549): Connection reset by peer (error 104)
2026-08-09T04:55:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2087 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:55:18 dist-test-slave-4tdj krb5kdc[2046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:55:18 dist-test-slave-4tdj krb5kdc[2046](info): set up 2 sockets
Aug 09 04:55:18 dist-test-slave-4tdj krb5kdc[2046](info): commencing operation
krb5kdc: starting...
W20260809 04:55:19.917732 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:55:20 dist-test-slave-4tdj krb5kdc[2046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:55:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:55:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:55:20.191236 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:37761
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:37761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:20.543220 2067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:20.543522 2067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:20.543673 2067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:20.550786 2067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:20.550909 2067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:20.551003 2067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:20.551107 2067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:20.551174 2067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:20.560637 2067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:37761
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:37761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:20.561846 2067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:20.563213 2067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:20.574753 2074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:20.574688 2081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:20.574931 2084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:20.575690 2067 server_base.cc:1034] running on GCE node
I20260809 04:55:20.576514 2067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:20.577849 2067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:20.579054 2067 hybrid_clock.cc:648] HybridClock initialized: now 1786251320579008 us; error 33 us; skew 500 ppm
I20260809 04:55:20.579396 2067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:20 dist-test-slave-4tdj krb5kdc[2046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251320, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:20.582549 2067 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:55:20.583931 2067 webserver.cc:460] Webserver started at http://127.31.93.62:40561/ using document root <none> and password file <none>
I20260809 04:55:20.584650 2067 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:20.584775 2067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:20.585088 2067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:20.588091 2067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "86aad4aff3894d66be871213ffe71518"
format_stamp: "Formatted at 2026-08-09 04:55:20 on dist-test-slave-4tdj"
server_key: "9e3fdb8f053873d4cdeceaefcc57d187"
server_key_iv: "d5264dc0e492e1e2d804ff4a660d1253"
server_key_version: "encryptionkey@0"
I20260809 04:55:20.588860 2067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "86aad4aff3894d66be871213ffe71518"
format_stamp: "Formatted at 2026-08-09 04:55:20 on dist-test-slave-4tdj"
server_key: "9e3fdb8f053873d4cdeceaefcc57d187"
server_key_iv: "d5264dc0e492e1e2d804ff4a660d1253"
server_key_version: "encryptionkey@0"
I20260809 04:55:20.593937 2067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 04:55:20.597369 2093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:20.598676 2067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:55:20.598826 2067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "86aad4aff3894d66be871213ffe71518"
format_stamp: "Formatted at 2026-08-09 04:55:20 on dist-test-slave-4tdj"
server_key: "9e3fdb8f053873d4cdeceaefcc57d187"
server_key_iv: "d5264dc0e492e1e2d804ff4a660d1253"
server_key_version: "encryptionkey@0"
I20260809 04:55:20.598989 2067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:20.631695 2067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:20.635628 2067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:20.635874 2067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:20.652578 2067 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:37761
I20260809 04:55:20.652621 2176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:37761 every 8 connection(s)
I20260809 04:55:20.653833 2067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:20.655313 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 2067
I20260809 04:55:20.655486 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:20.656055 32116 external_mini_cluster.cc:1428] Setting key b415f1a52f1259fee7c6c0c5e67dfbad
I20260809 04:55:20.658071 2178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:20.668749 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:20.657779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44117 (local address 127.31.93.62:37761)
0809 04:55:20.658528 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:20.658538 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:20.658595 (+ 57us) server_negotiation.cc:407] Connection header received
0809 04:55:20.660153 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:20.660173 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:20.660528 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:20.660849 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:20.661381 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:20.662739 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:20.664587 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:20.665192 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:20.665324 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:20.666604 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:20.666612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:20.666624 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:20.667131 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:20.667204 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:20.667230 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:20.667358 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:20.667644 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 04:55:20.667816 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":320,"threads_started":1}
I20260809 04:55:20.670138 2178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518: Bootstrap starting.
I20260809 04:55:20.672358 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:20.673241 2178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:20.674182 2178 log.cc:824] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:20.676909 2178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518: No bootstrap required, opened a new log
I20260809 04:55:20.682147 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:20.676422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53989 (local address 127.31.93.62:37761)
0809 04:55:20.676583 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:20.676588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:20.677075 (+ 487us) server_negotiation.cc:407] Connection header received
0809 04:55:20.677205 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:20.677209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:20.677267 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:20.677353 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:20.677811 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:20.678612 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:20.679868 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:20.680307 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:20.680422 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:20.681454 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:20.681458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:20.681461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:20.681480 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:20.681518 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:20.681532 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:20.681632 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:20.681842 (+ 210us) server_negotiation.cc:299] Negotiation successful
0809 04:55:20.681893 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 04:55:20.684333 2178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86aad4aff3894d66be871213ffe71518" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37761 } }
I20260809 04:55:20.684676 2178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:55:20.684764 2178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86aad4aff3894d66be871213ffe71518, State: Initialized, Role: FOLLOWER
I20260809 04:55:20.685281 2178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86aad4aff3894d66be871213ffe71518" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37761 } }
I20260809 04:55:20.685505 2178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:55:20.685638 2178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:55:20.685792 2178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:20.688202 2178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86aad4aff3894d66be871213ffe71518" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37761 } }
I20260809 04:55:20.688660 2178 leader_election.cc:302] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86aad4aff3894d66be871213ffe71518; no voters:
I20260809 04:55:20.689186 2178 leader_election.cc:288] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:55:20.689312 2193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:20.689875 2193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [term 1 LEADER]: Becoming Leader. State: Replica: 86aad4aff3894d66be871213ffe71518, State: Running, Role: LEADER
I20260809 04:55:20.690402 2193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86aad4aff3894d66be871213ffe71518" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37761 } }
I20260809 04:55:20.691066 2178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:20.693521 2194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86aad4aff3894d66be871213ffe71518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86aad4aff3894d66be871213ffe71518" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37761 } } }
I20260809 04:55:20.693899 2194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:20.698180 2196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86aad4aff3894d66be871213ffe71518. Latest consensus state: current_term: 1 leader_uuid: "86aad4aff3894d66be871213ffe71518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86aad4aff3894d66be871213ffe71518" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37761 } } }
I20260809 04:55:20.698498 2196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:20.699115 2206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:20.703855 2206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:55:20.711664 2206 catalog_manager.cc:1347] Generated new cluster ID: 553f2b487e1a43c18c2a75e50ec2e7e5
I20260809 04:55:20.711767 2206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:20.721519 2206 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:20.723346 2206 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:20.730914 2206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86aad4aff3894d66be871213ffe71518: Generated new TSK 0
I20260809 04:55:20.731865 2206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:20.741767 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 2067
W20260809 04:55:20.742868 2169 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44117
W20260809 04:55:20.743661 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 2067: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:55:20.743789 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 2067
W20260809 04:55:20.765406 2042 connection.cc:535] client connection to 127.31.93.62:37761 recv error: Network error: failed to read from TLS socket (remote: 127.31.93.62:37761): Connection reset by peer (error 104)
2026-08-09T04:55:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2038 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:55:20 dist-test-slave-4tdj krb5kdc[2233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:55:20 dist-test-slave-4tdj krb5kdc[2233](info): set up 2 sockets
Aug 09 04:55:20 dist-test-slave-4tdj krb5kdc[2233](info): commencing operation
krb5kdc: starting...
W20260809 04:55:21.949867 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:55:22 dist-test-slave-4tdj krb5kdc[2233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:55:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:55:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:55:22.211079 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:37469
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:39487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:37469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:22.529721 2249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:22.529973 2249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:22.530102 2249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:22.537027 2249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:22.537120 2249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:22.537189 2249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:22.537269 2249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:22.537328 2249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:22.546669 2249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:39487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:37469
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:37469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:22.547869 2249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:22.549146 2249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:22.560235 2262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:22.560348 2259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:22.560357 2264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:22.561091 2249 server_base.cc:1034] running on GCE node
I20260809 04:55:22.561794 2249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:22.563176 2249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:22.564401 2249 hybrid_clock.cc:648] HybridClock initialized: now 1786251322564338 us; error 40 us; skew 500 ppm
I20260809 04:55:22.564738 2249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:22 dist-test-slave-4tdj krb5kdc[2233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251322, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:22.570992 2249 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:55:22.572328 2249 webserver.cc:460] Webserver started at http://127.31.93.62:45941/ using document root <none> and password file <none>
I20260809 04:55:22.572983 2249 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:22.573109 2249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:22.573387 2249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:22.576248 2249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "e37a41d518674dfda8365788a5779263"
format_stamp: "Formatted at 2026-08-09 04:55:22 on dist-test-slave-4tdj"
server_key: "de3cfa7de52775f9bec40ede13bc42f0"
server_key_iv: "ef0619492698277ee1c669578dbe4653"
server_key_version: "encryptionkey@0"
I20260809 04:55:22.576977 2249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "e37a41d518674dfda8365788a5779263"
format_stamp: "Formatted at 2026-08-09 04:55:22 on dist-test-slave-4tdj"
server_key: "de3cfa7de52775f9bec40ede13bc42f0"
server_key_iv: "ef0619492698277ee1c669578dbe4653"
server_key_version: "encryptionkey@0"
I20260809 04:55:22.581869 2249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260809 04:55:22.585197 2276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:22.586527 2249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:55:22.586685 2249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "e37a41d518674dfda8365788a5779263"
format_stamp: "Formatted at 2026-08-09 04:55:22 on dist-test-slave-4tdj"
server_key: "de3cfa7de52775f9bec40ede13bc42f0"
server_key_iv: "ef0619492698277ee1c669578dbe4653"
server_key_version: "encryptionkey@0"
I20260809 04:55:22.586859 2249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:22.592360 2249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:22.596074 2249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:22.596345 2249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:22.612753 2249 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:37469
I20260809 04:55:22.612789 2363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:37469 every 8 connection(s)
I20260809 04:55:22.614035 2249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:22.617532 2365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:22.623642 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 2249
I20260809 04:55:22.623803 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:22.624243 32116 external_mini_cluster.cc:1428] Setting key f416d057cf0d5fd394ee24f4399668da
I20260809 04:55:22.627518 2365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263: Bootstrap starting.
I20260809 04:55:22.630801 2365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:22.631604 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:22.625868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38663 (local address 127.31.93.62:37469)
0809 04:55:22.626654 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:22.626665 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:22.626707 (+ 42us) server_negotiation.cc:407] Connection header received
0809 04:55:22.628316 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:22.628338 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:22.628650 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:22.628962 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:22.630000 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:22.630006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:22.630019 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:22.630035 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:22.630094 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:22.630114 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:22.630264 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:22.630523 (+ 259us) server_negotiation.cc:299] Negotiation successful
0809 04:55:22.630720 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":358,"threads_started":1}
I20260809 04:55:22.631851 2365 log.cc:824] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:22.634835 2365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263: No bootstrap required, opened a new log
I20260809 04:55:22.635283 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:22.642413 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:22.640112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43001 (local address 127.31.93.62:37469)
0809 04:55:22.640271 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:22.640277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:22.640717 (+ 440us) server_negotiation.cc:407] Connection header received
0809 04:55:22.640826 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:22.640831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:22.640888 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:22.640989 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:22.641820 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:22.641826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:22.641831 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:22.641848 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:22.641889 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:22.641912 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:22.642031 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:22.642086 (+ 55us) server_negotiation.cc:299] Negotiation successful
0809 04:55:22.642134 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 04:55:22.643996 2365 raft_consensus.cc:348] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37a41d518674dfda8365788a5779263" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37469 } }
I20260809 04:55:22.644301 2365 raft_consensus.cc:374] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:55:22.644371 2365 raft_consensus.cc:729] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e37a41d518674dfda8365788a5779263, State: Initialized, Role: FOLLOWER
I20260809 04:55:22.644850 2365 consensus_queue.cc:260] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37a41d518674dfda8365788a5779263" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37469 } }
I20260809 04:55:22.645061 2365 raft_consensus.cc:388] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:55:22.645177 2365 raft_consensus.cc:482] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:55:22.645329 2365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:22.647435 2365 raft_consensus.cc:504] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37a41d518674dfda8365788a5779263" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37469 } }
I20260809 04:55:22.647846 2365 leader_election.cc:302] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e37a41d518674dfda8365788a5779263; no voters:
I20260809 04:55:22.648322 2365 leader_election.cc:288] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:55:22.648440 2379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:22.649096 2379 raft_consensus.cc:686] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [term 1 LEADER]: Becoming Leader. State: Replica: e37a41d518674dfda8365788a5779263, State: Running, Role: LEADER
I20260809 04:55:22.649600 2379 consensus_queue.cc:237] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37a41d518674dfda8365788a5779263" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37469 } }
I20260809 04:55:22.650183 2365 sys_catalog.cc:564] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:22.652246 2386 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e37a41d518674dfda8365788a5779263. Latest consensus state: current_term: 1 leader_uuid: "e37a41d518674dfda8365788a5779263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37a41d518674dfda8365788a5779263" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37469 } } }
I20260809 04:55:22.652637 2386 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:22.653163 2381 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e37a41d518674dfda8365788a5779263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37a41d518674dfda8365788a5779263" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 37469 } } }
I20260809 04:55:22.653455 2381 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:22.658808 2392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:22.663386 2392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:55:22.671159 2392 catalog_manager.cc:1347] Generated new cluster ID: 7db272da70a540f48e7c58fe9b54672e
I20260809 04:55:22.671259 2392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:22.684842 2392 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:22.685200 2392 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:22.692021 2392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e37a41d518674dfda8365788a5779263: Generated new TSK 0
I20260809 04:55:22.692881 2392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:22.702210 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 2249
W20260809 04:55:22.703297 2358 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38663
W20260809 04:55:22.703991 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 2249: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:55:22.704128 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 2249
2026-08-09T04:55:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1968 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:55:22 dist-test-slave-4tdj krb5kdc[2422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:55:22 dist-test-slave-4tdj krb5kdc[2422](info): set up 2 sockets
Aug 09 04:55:22 dist-test-slave-4tdj krb5kdc[2422](info): commencing operation
krb5kdc: starting...
W20260809 04:55:23.931632 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.001s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:55:24 dist-test-slave-4tdj krb5kdc[2422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:55:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:55:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:55:24.186851 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:44699
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:43145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:44699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:24.505247 2440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:24.505508 2440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:24.505653 2440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:24.512540 2440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:24.512643 2440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:24.512720 2440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:24.512809 2440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:24.512866 2440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:24.522099 2440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:43145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:44699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:44699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:24.523252 2440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:24.524498 2440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:24.535539 2446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:24.535586 2448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:24.535672 2455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:24.536373 2440 server_base.cc:1034] running on GCE node
I20260809 04:55:24.537067 2440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:24.538321 2440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:24.539522 2440 hybrid_clock.cc:648] HybridClock initialized: now 1786251324539478 us; error 36 us; skew 500 ppm
I20260809 04:55:24.539840 2440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:24 dist-test-slave-4tdj krb5kdc[2422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251324, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:24.542686 2440 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:55:24.544054 2440 webserver.cc:460] Webserver started at http://127.31.93.62:40775/ using document root <none> and password file <none>
I20260809 04:55:24.544667 2440 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:24.544783 2440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:24.545058 2440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:24.547873 2440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "8bce60e46bc04fedb7bba78284790db6"
format_stamp: "Formatted at 2026-08-09 04:55:24 on dist-test-slave-4tdj"
server_key: "8a6a224e77c90db39bc1522fd36d618f"
server_key_iv: "3dbd69d3b302ba03a4da67ecd28d5974"
server_key_version: "encryptionkey@0"
I20260809 04:55:24.548588 2440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "8bce60e46bc04fedb7bba78284790db6"
format_stamp: "Formatted at 2026-08-09 04:55:24 on dist-test-slave-4tdj"
server_key: "8a6a224e77c90db39bc1522fd36d618f"
server_key_iv: "3dbd69d3b302ba03a4da67ecd28d5974"
server_key_version: "encryptionkey@0"
I20260809 04:55:24.553340 2440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260809 04:55:24.556515 2471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:24.557725 2440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:55:24.557875 2440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "8bce60e46bc04fedb7bba78284790db6"
format_stamp: "Formatted at 2026-08-09 04:55:24 on dist-test-slave-4tdj"
server_key: "8a6a224e77c90db39bc1522fd36d618f"
server_key_iv: "3dbd69d3b302ba03a4da67ecd28d5974"
server_key_version: "encryptionkey@0"
I20260809 04:55:24.558037 2440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:24.582283 2440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:24.585913 2440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:24.586160 2440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:24.602959 2440 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:44699
I20260809 04:55:24.603003 2548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:44699 every 8 connection(s)
I20260809 04:55:24.604205 2440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:24.607654 2549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:24.608912 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 2440
I20260809 04:55:24.609094 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:24.609663 32116 external_mini_cluster.cc:1428] Setting key a04008645de32799b1eb7805f9474ba5
I20260809 04:55:24.619590 2549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6: Bootstrap starting.
Aug 09 04:55:24 dist-test-slave-4tdj krb5kdc[2422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:55:24.622648 2549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:24.623847 2549 log.cc:824] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:24.626718 2549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6: No bootstrap required, opened a new log
I20260809 04:55:24.630525 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:24.611512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36895 (local address 127.31.93.62:44699)
0809 04:55:24.612225 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:24.612236 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:24.612274 (+ 38us) server_negotiation.cc:407] Connection header received
0809 04:55:24.613853 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:24.613874 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:24.614244 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:24.614626 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:24.615980 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:24.617271 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:24.618586 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:24.619207 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:24.619369 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:24.623602 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:24.623610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:24.623622 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:24.623655 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:24.626788 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:24.627403 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:24.627409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:24.627415 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:24.627582 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:24.627954 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:24.627958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:24.627960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:24.628452 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:24.628761 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:24.629166 (+ 405us) server_negotiation.cc:299] Negotiation successful
0809 04:55:24.629542 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":312,"threads_started":1}
I20260809 04:55:24.633926 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:24.634320 2549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bce60e46bc04fedb7bba78284790db6" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44699 } }
I20260809 04:55:24.634698 2549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:55:24.634804 2549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bce60e46bc04fedb7bba78284790db6, State: Initialized, Role: FOLLOWER
I20260809 04:55:24.635368 2549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bce60e46bc04fedb7bba78284790db6" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44699 } }
I20260809 04:55:24.635618 2549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:55:24.635746 2549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:55:24.635883 2549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:24.638752 2549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bce60e46bc04fedb7bba78284790db6" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44699 } }
I20260809 04:55:24.639264 2549 leader_election.cc:302] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bce60e46bc04fedb7bba78284790db6; no voters:
I20260809 04:55:24.639894 2549 leader_election.cc:288] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:55:24.639971 2568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:24.641968 2549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:24.646581 2568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [term 1 LEADER]: Becoming Leader. State: Replica: 8bce60e46bc04fedb7bba78284790db6, State: Running, Role: LEADER
I20260809 04:55:24.647243 2568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bce60e46bc04fedb7bba78284790db6" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44699 } }
I20260809 04:55:24.650604 2573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bce60e46bc04fedb7bba78284790db6. Latest consensus state: current_term: 1 leader_uuid: "8bce60e46bc04fedb7bba78284790db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bce60e46bc04fedb7bba78284790db6" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44699 } } }
I20260809 04:55:24.651050 2573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:24.650573 2569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bce60e46bc04fedb7bba78284790db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bce60e46bc04fedb7bba78284790db6" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 44699 } } }
I20260809 04:55:24.652063 2583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:24.652951 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:24.638675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41045 (local address 127.31.93.62:44699)
0809 04:55:24.638846 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:24.638852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:24.639358 (+ 506us) server_negotiation.cc:407] Connection header received
0809 04:55:24.639527 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:24.639531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:24.639589 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:24.639671 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:24.641343 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:24.642174 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:24.643523 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:24.643964 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:24.644078 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:24.646187 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:24.646192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:24.646195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:24.646223 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:24.649496 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:24.650329 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:24.650335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:24.650339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:24.650471 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:24.650955 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:24.650959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:24.650963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:24.651429 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:24.652419 (+ 990us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:24.652528 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 04:55:24.652668 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:55:24.652957 2569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:24.659929 2583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:55:24.671808 2583 catalog_manager.cc:1347] Generated new cluster ID: 84efd5e738f643f280260942cb1bb34b
I20260809 04:55:24.671895 2583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:24.703130 2583 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:24.704689 2583 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:24.713613 2583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bce60e46bc04fedb7bba78284790db6: Generated new TSK 0
I20260809 04:55:24.714461 2583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:24.727427 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 2440
I20260809 04:55:24.728343 2543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:55:24.818709 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 2440
2026-08-09T04:55:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2110 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:55:24 dist-test-slave-4tdj krb5kdc[2606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:55:24 dist-test-slave-4tdj krb5kdc[2606](info): set up 2 sockets
Aug 09 04:55:24 dist-test-slave-4tdj krb5kdc[2606](info): commencing operation
krb5kdc: starting...
W20260809 04:55:26.053253 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:55:26 dist-test-slave-4tdj krb5kdc[2606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:55:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:55:26.305882 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:45837
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:45837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:26.621151 2640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:26.621413 2640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:26.621517 2640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:26.628396 2640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:26.628499 2640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:26.628552 2640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:26.628612 2640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:26.628654 2640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:26.638006 2640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:45837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:45837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:26.639118 2640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:26.640309 2640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:26.651396 2648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:26.651590 2651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:26.651666 2649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:26.652259 2640 server_base.cc:1034] running on GCE node
I20260809 04:55:26.653043 2640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:26.654423 2640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:26.655627 2640 hybrid_clock.cc:648] HybridClock initialized: now 1786251326655569 us; error 47 us; skew 500 ppm
I20260809 04:55:26.655963 2640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:26 dist-test-slave-4tdj krb5kdc[2606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251326, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:26.659080 2640 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:55:26.660491 2640 webserver.cc:460] Webserver started at http://127.31.93.62:34857/ using document root <none> and password file <none>
I20260809 04:55:26.661161 2640 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:26.661293 2640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:26.661586 2640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:26.664623 2640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "3a208b06370b4552b7e9ef7c08df1d63"
format_stamp: "Formatted at 2026-08-09 04:55:26 on dist-test-slave-4tdj"
server_key: "833e8957dd28a2a6df131ed20be3ead4"
server_key_iv: "73503a88f025ae7918f103970900f603"
server_key_version: "encryptionkey@0"
I20260809 04:55:26.665411 2640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "3a208b06370b4552b7e9ef7c08df1d63"
format_stamp: "Formatted at 2026-08-09 04:55:26 on dist-test-slave-4tdj"
server_key: "833e8957dd28a2a6df131ed20be3ead4"
server_key_iv: "73503a88f025ae7918f103970900f603"
server_key_version: "encryptionkey@0"
I20260809 04:55:26.670718 2640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260809 04:55:26.674070 2665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:26.675364 2640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 04:55:26.675537 2640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "3a208b06370b4552b7e9ef7c08df1d63"
format_stamp: "Formatted at 2026-08-09 04:55:26 on dist-test-slave-4tdj"
server_key: "833e8957dd28a2a6df131ed20be3ead4"
server_key_iv: "73503a88f025ae7918f103970900f603"
server_key_version: "encryptionkey@0"
I20260809 04:55:26.675717 2640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:26.686465 2640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:26.690685 2640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:26.690953 2640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:26.708715 2640 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:45837
I20260809 04:55:26.708745 2757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:45837 every 8 connection(s)
I20260809 04:55:26.710062 2640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:26.713460 2759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:26.716804 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 2640
I20260809 04:55:26.716987 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:26.717423 32116 external_mini_cluster.cc:1428] Setting key a914a37df702888cf53934f821c9c0fe
I20260809 04:55:26.726012 2759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63: Bootstrap starting.
I20260809 04:55:26.729517 2759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:26.730846 2759 log.cc:824] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:26.731215 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:26.719154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39025 (local address 127.31.93.62:45837)
0809 04:55:26.719976 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:26.719992 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:26.720039 (+ 47us) server_negotiation.cc:407] Connection header received
0809 04:55:26.721755 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:26.721782 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:26.722278 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:26.722697 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:26.723193 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:26.725130 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:26.726496 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:26.727466 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:26.727672 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:26.728646 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:26.728657 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:26.728673 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:26.729322 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:26.729415 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:26.729446 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:26.729564 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:26.729697 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 04:55:26.729949 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":350,"threads_started":1}
I20260809 04:55:26.733645 2759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63: No bootstrap required, opened a new log
I20260809 04:55:26.735049 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:26.743238 2759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a208b06370b4552b7e9ef7c08df1d63" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45837 } }
I20260809 04:55:26.743623 2759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:55:26.743729 2759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a208b06370b4552b7e9ef7c08df1d63, State: Initialized, Role: FOLLOWER
I20260809 04:55:26.744277 2759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a208b06370b4552b7e9ef7c08df1d63" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45837 } }
I20260809 04:55:26.744513 2759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:55:26.744647 2759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:55:26.744859 2759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:26.746459 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:26.740082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58179 (local address 127.31.93.62:45837)
0809 04:55:26.740256 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:26.740263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:26.740707 (+ 444us) server_negotiation.cc:407] Connection header received
0809 04:55:26.740854 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:26.740857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:26.740910 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:26.740984 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:26.741547 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:26.742223 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:26.744025 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:26.744563 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:26.744697 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:26.745710 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:26.745715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:26.745718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:26.745739 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:26.745783 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:26.745800 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:26.745885 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:26.746019 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 04:55:26.746089 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 04:55:26.747558 2759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a208b06370b4552b7e9ef7c08df1d63" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45837 } }
I20260809 04:55:26.748047 2759 leader_election.cc:302] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a208b06370b4552b7e9ef7c08df1d63; no voters:
I20260809 04:55:26.748544 2759 leader_election.cc:288] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:55:26.748713 2775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:26.749387 2775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [term 1 LEADER]: Becoming Leader. State: Replica: 3a208b06370b4552b7e9ef7c08df1d63, State: Running, Role: LEADER
I20260809 04:55:26.749904 2775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a208b06370b4552b7e9ef7c08df1d63" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45837 } }
I20260809 04:55:26.750499 2759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:26.752447 2777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a208b06370b4552b7e9ef7c08df1d63. Latest consensus state: current_term: 1 leader_uuid: "3a208b06370b4552b7e9ef7c08df1d63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a208b06370b4552b7e9ef7c08df1d63" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45837 } } }
I20260809 04:55:26.752806 2777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:26.753539 2776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a208b06370b4552b7e9ef7c08df1d63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a208b06370b4552b7e9ef7c08df1d63" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45837 } } }
I20260809 04:55:26.753821 2776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:26.754379 2789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:26.760351 2789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:55:26.769773 2789 catalog_manager.cc:1347] Generated new cluster ID: 39f1b4e5d8c04c5b9ee4d2cdb199909a
I20260809 04:55:26.769871 2789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:26.781265 2789 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:26.782456 2789 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:26.797919 2789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a208b06370b4552b7e9ef7c08df1d63: Generated new TSK 0
I20260809 04:55:26.798696 2789 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:26.813761 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 2640
W20260809 04:55:26.814941 2751 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39025
W20260809 04:55:26.815639 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 2640: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:55:26.815761 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 2640
2026-08-09T04:55:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1994 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:55:26 dist-test-slave-4tdj krb5kdc[2815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:55:26 dist-test-slave-4tdj krb5kdc[2815](info): set up 2 sockets
Aug 09 04:55:26 dist-test-slave-4tdj krb5kdc[2815](info): commencing operation
krb5kdc: starting...
W20260809 04:55:28.021492 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:55:28 dist-test-slave-4tdj krb5kdc[2815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:55:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:55:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:55:28.279346 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:45877
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:43157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:45877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:28.599781 2840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:28.600039 2840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:28.600128 2840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:28.607077 2840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:28.607172 2840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:28.607226 2840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:28.607291 2840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:28.607334 2840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:28.616648 2840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:43157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:45877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:45877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:28.617797 2840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:28.619065 2840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:28.629948 2847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:28.630038 2850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:28.630180 2854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:28.630918 2840 server_base.cc:1034] running on GCE node
I20260809 04:55:28.631650 2840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:28.632891 2840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:28.634083 2840 hybrid_clock.cc:648] HybridClock initialized: now 1786251328634039 us; error 36 us; skew 500 ppm
I20260809 04:55:28.634425 2840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:28 dist-test-slave-4tdj krb5kdc[2815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251328, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:28.640254 2840 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:55:28.641726 2840 webserver.cc:460] Webserver started at http://127.31.93.62:44207/ using document root <none> and password file <none>
I20260809 04:55:28.642405 2840 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:28.642532 2840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:28.642815 2840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:28.645648 2840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "40c0f2f2cf614c9fb6387dc8a6e73083"
format_stamp: "Formatted at 2026-08-09 04:55:28 on dist-test-slave-4tdj"
server_key: "e24cafc8bf13a14e3e40a3a80bb859cf"
server_key_iv: "d4e99b1ffa4d9c85011c4e377d6afa40"
server_key_version: "encryptionkey@0"
I20260809 04:55:28.646406 2840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "40c0f2f2cf614c9fb6387dc8a6e73083"
format_stamp: "Formatted at 2026-08-09 04:55:28 on dist-test-slave-4tdj"
server_key: "e24cafc8bf13a14e3e40a3a80bb859cf"
server_key_iv: "d4e99b1ffa4d9c85011c4e377d6afa40"
server_key_version: "encryptionkey@0"
I20260809 04:55:28.651124 2840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 04:55:28.654335 2871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:28.655541 2840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:55:28.655700 2840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "40c0f2f2cf614c9fb6387dc8a6e73083"
format_stamp: "Formatted at 2026-08-09 04:55:28 on dist-test-slave-4tdj"
server_key: "e24cafc8bf13a14e3e40a3a80bb859cf"
server_key_iv: "d4e99b1ffa4d9c85011c4e377d6afa40"
server_key_version: "encryptionkey@0"
I20260809 04:55:28.655872 2840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:28.661213 2840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:28.665150 2840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:28.665422 2840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:28.682003 2840 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:45877
I20260809 04:55:28.682032 2950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:45877 every 8 connection(s)
I20260809 04:55:28.683274 2840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:28.686717 2951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:28.692296 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 2840
I20260809 04:55:28.692458 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:28.692862 32116 external_mini_cluster.cc:1428] Setting key c86685e295398b64146a8982219273e5
I20260809 04:55:28.697540 2951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083: Bootstrap starting.
I20260809 04:55:28.701309 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:28.694596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49785 (local address 127.31.93.62:45877)
0809 04:55:28.695447 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:28.695461 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:28.695508 (+ 47us) server_negotiation.cc:407] Connection header received
0809 04:55:28.697690 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:28.697715 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:28.698113 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:28.698522 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:28.699482 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:28.699494 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:28.699512 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:28.699529 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:28.699622 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:28.699651 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:28.699865 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:28.699987 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 04:55:28.700163 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":341,"threads_started":1}
I20260809 04:55:28.701388 2951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:28.702486 2951 log.cc:824] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:28.704869 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:28.705266 2951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083: No bootstrap required, opened a new log
I20260809 04:55:28.711040 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:28.708889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48501 (local address 127.31.93.62:45877)
0809 04:55:28.709084 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:28.709090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:28.709377 (+ 287us) server_negotiation.cc:407] Connection header received
0809 04:55:28.709496 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:28.709501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:28.709556 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:28.709650 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:28.710438 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:28.710444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:28.710449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 04:55:28.710465 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:28.710517 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 04:55:28.710535 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:28.710654 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:28.710696 (+ 42us) server_negotiation.cc:299] Negotiation successful
0809 04:55:28.710752 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 04:55:28.713038 2951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c0f2f2cf614c9fb6387dc8a6e73083" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45877 } }
I20260809 04:55:28.713346 2951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 04:55:28.713418 2951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c0f2f2cf614c9fb6387dc8a6e73083, State: Initialized, Role: FOLLOWER
I20260809 04:55:28.713900 2951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c0f2f2cf614c9fb6387dc8a6e73083" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45877 } }
I20260809 04:55:28.714104 2951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 04:55:28.714198 2951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 04:55:28.714354 2951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:28.716400 2951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c0f2f2cf614c9fb6387dc8a6e73083" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45877 } }
I20260809 04:55:28.716822 2951 leader_election.cc:302] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40c0f2f2cf614c9fb6387dc8a6e73083; no voters:
I20260809 04:55:28.717329 2951 leader_election.cc:288] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 04:55:28.717438 2966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:28.718328 2966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [term 1 LEADER]: Becoming Leader. State: Replica: 40c0f2f2cf614c9fb6387dc8a6e73083, State: Running, Role: LEADER
I20260809 04:55:28.718910 2966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c0f2f2cf614c9fb6387dc8a6e73083" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45877 } }
I20260809 04:55:28.719316 2951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:28.721596 2969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40c0f2f2cf614c9fb6387dc8a6e73083. Latest consensus state: current_term: 1 leader_uuid: "40c0f2f2cf614c9fb6387dc8a6e73083" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c0f2f2cf614c9fb6387dc8a6e73083" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45877 } } }
I20260809 04:55:28.721740 2967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40c0f2f2cf614c9fb6387dc8a6e73083" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c0f2f2cf614c9fb6387dc8a6e73083" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45877 } } }
I20260809 04:55:28.721995 2969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:28.722055 2967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083 [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:28.723683 2980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:28.727970 2980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 04:55:28.736272 2980 catalog_manager.cc:1347] Generated new cluster ID: a16b9347ac69416bb5d8a581ddc826e1
I20260809 04:55:28.736407 2980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:28.767320 2980 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:28.767827 2980 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:28.779309 2980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40c0f2f2cf614c9fb6387dc8a6e73083: Generated new TSK 0
I20260809 04:55:28.780308 2980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:28.786443 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 2840
W20260809 04:55:28.787418 2941 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49785
W20260809 04:55:28.788116 32116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 2840: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 04:55:28.788242 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 2840
2026-08-09T04:55:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1971 ms)
[----------] 18 tests from AuthTokenIssuingTest (36393 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:55:28 dist-test-slave-4tdj krb5kdc[2998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:55:28 dist-test-slave-4tdj krb5kdc[2998](info): set up 2 sockets
Aug 09 04:55:28 dist-test-slave-4tdj krb5kdc[2998](info): commencing operation
krb5kdc: starting...
W20260809 04:55:30.033133 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.002s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:55:30 dist-test-slave-4tdj krb5kdc[2998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:55:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:55:30.292834 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:45259
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:45259,127.31.93.61:34263,127.31.93.60:44579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:30.612366 3019 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 04:55:30.612661 3019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:30.612733 3019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:30.612810 3019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:30.619738 3019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:30.619822 3019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:30.619884 3019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:30.619931 3019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:30.619990 3019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:30.629503 3019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:45259,127.31.93.61:34263,127.31.93.60:44579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:45259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:30.630710 3019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:30.631955 3019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:30.643033 3024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:30.642958 3029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:30.642999 3027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:30.643942 3019 server_base.cc:1034] running on GCE node
I20260809 04:55:30.644743 3019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:30.646047 3019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:30.647267 3019 hybrid_clock.cc:648] HybridClock initialized: now 1786251330647207 us; error 33 us; skew 500 ppm
I20260809 04:55:30.647569 3019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:30 dist-test-slave-4tdj krb5kdc[2998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251330, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:30.650446 3019 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:55:30.651804 3019 webserver.cc:460] Webserver started at http://127.31.93.62:41847/ using document root <none> and password file <none>
I20260809 04:55:30.652422 3019 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:30.652520 3019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:30.652813 3019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:30.655679 3019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "0a61f438e7094124afa4a3fab2c2a07f"
format_stamp: "Formatted at 2026-08-09 04:55:30 on dist-test-slave-4tdj"
server_key: "17429fe42e8206b8ea80942ffaec3b8c"
server_key_iv: "ddc66e05e69ce1c9a87d13341f397acd"
server_key_version: "encryptionkey@0"
I20260809 04:55:30.656427 3019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "0a61f438e7094124afa4a3fab2c2a07f"
format_stamp: "Formatted at 2026-08-09 04:55:30 on dist-test-slave-4tdj"
server_key: "17429fe42e8206b8ea80942ffaec3b8c"
server_key_iv: "ddc66e05e69ce1c9a87d13341f397acd"
server_key_version: "encryptionkey@0"
I20260809 04:55:30.661108 3019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 04:55:30.664301 3042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:30.665475 3019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 04:55:30.665630 3019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "0a61f438e7094124afa4a3fab2c2a07f"
format_stamp: "Formatted at 2026-08-09 04:55:30 on dist-test-slave-4tdj"
server_key: "17429fe42e8206b8ea80942ffaec3b8c"
server_key_iv: "ddc66e05e69ce1c9a87d13341f397acd"
server_key_version: "encryptionkey@0"
I20260809 04:55:30.665817 3019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:30.688432 3019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:30.692035 3019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:30.692286 3019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:30.709000 3019 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:45259
I20260809 04:55:30.709019 3132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:45259 every 8 connection(s)
I20260809 04:55:30.710235 3019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:30.713723 3134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:30.716662 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 3019
I20260809 04:55:30.716902 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:30.717846 32116 external_mini_cluster.cc:1428] Setting key 3d68b5ce04a82c92c0aabe05d0c611a6
I20260809 04:55:30.721767 3134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:55:30 dist-test-slave-4tdj krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251330, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:55:30.743557 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:30.725113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45259 (local address 127.0.0.1:60774)
0809 04:55:30.725911 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:30.725965 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:30.727931 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:30.728716 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:30.728722 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:30.729380 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:30.729715 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:30.729732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:30.731345 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:30.731349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:30.732393 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:30.732404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:30.733531 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:30.733536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:30.733745 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:30.734358 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:30.734380 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:30.736402 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:30.739990 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:30.740000 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:30.740008 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:30.740378 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:30.740796 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:30.740799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:30.740801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:30.740947 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:30.741805 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:30.741810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:30.741977 (+ 167us) client_negotiation.cc:769] Sending connection context
0809 04:55:30.742167 (+ 190us) client_negotiation.cc:240] Negotiation successful
0809 04:55:30.742548 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":43,"thread_start_us":320,"threads_started":1}
I20260809 04:55:30.743593 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:30.726359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60774 (local address 127.31.93.62:45259)
0809 04:55:30.726991 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:30.726998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:30.727053 (+ 55us) server_negotiation.cc:407] Connection header received
0809 04:55:30.728287 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:30.728292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:30.728505 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:30.728677 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:30.729903 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:30.731169 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:30.732531 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:30.733328 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:30.733479 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:30.736618 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:30.736628 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:30.736641 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:30.736678 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:30.739876 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:30.740503 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:30.740511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:30.740514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:30.740696 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:30.741049 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:30.741053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:30.741057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:30.741671 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:30.741856 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:30.742350 (+ 494us) server_negotiation.cc:299] Negotiation successful
0809 04:55:30.742717 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":430,"thread_start_us":277,"threads_started":1}
I20260809 04:55:30.746692 3134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:55:30.747777 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:30.747422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:34263 (local address 127.0.0.1:36744)
0809 04:55:30.747571 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:30.747669 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:34263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 04:55:30.748049 3049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.93.61:34263: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.31.93.61@KRBTEST.COM; defaulting to no policy
I20260809 04:55:30.749505 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:30.749172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:34263 (local address 127.0.0.1:36760)
0809 04:55:30.749317 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:30.749379 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:34263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 04:55:30.749703 3134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.61:34263: Network error: Client connection negotiation failed: client connection to 127.31.93.61:34263: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.61.keytab.
Entry for principal kudu/127.31.93.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.61.keytab.
I20260809 04:55:30.780776 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.61:34263
--webserver_interface=127.31.93.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.61.keytab
--principal=kudu/127.31.93.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:45259,127.31.93.61:34263,127.31.93.60:44579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:55:30.799026 3134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } attempt: 1
I20260809 04:55:30.800228 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:30.799791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:34263 (local address 127.0.0.1:36762)
0809 04:55:30.799985 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:30.800096 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:34263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 04:55:30.801219 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:30.800863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:34263 (local address 127.0.0.1:36764)
0809 04:55:30.801012 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:30.801084 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:34263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 04:55:30.801411 3134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.61:34263: Network error: Client connection negotiation failed: client connection to 127.31.93.61:34263: connect: Connection refused (error 111)
I20260809 04:55:30.869645 3134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } attempt: 2
I20260809 04:55:30.871047 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:30.870443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:34263 (local address 127.0.0.1:36772)
0809 04:55:30.870758 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:30.870870 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:34263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260809 04:55:30.872309 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:30.871949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:34263 (local address 127.0.0.1:36778)
0809 04:55:30.872082 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:30.872170 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:34263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260809 04:55:30.872568 3134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.61:34263: Network error: Client connection negotiation failed: client connection to 127.31.93.61:34263: connect: Connection refused (error 111)
I20260809 04:55:30.963842 3134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } attempt: 3
I20260809 04:55:30.965039 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:30.964481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:34263 (local address 127.0.0.1:36784)
0809 04:55:30.964736 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:30.964857 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:34263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260809 04:55:30.966084 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:30.965676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:34263 (local address 127.0.0.1:36786)
0809 04:55:30.965822 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:30.965911 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:34263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 04:55:30.966313 3134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.61:34263: Network error: Client connection negotiation failed: client connection to 127.31.93.61:34263: connect: Connection refused (error 111)
W20260809 04:55:31.107417 3149 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 04:55:31.107720 3149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:31.107800 3149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:31.107883 3149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 04:55:31.109520 3134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } attempt: 4
I20260809 04:55:31.110731 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.110183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:34263 (local address 127.0.0.1:36788)
0809 04:55:31.110507 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.110595 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:34263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260809 04:55:31.111663 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.111363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:34263 (local address 127.0.0.1:36794)
0809 04:55:31.111474 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.111551 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:34263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260809 04:55:31.111881 3134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.61:34263: Network error: Client connection negotiation failed: client connection to 127.31.93.61:34263: connect: Connection refused (error 111)
W20260809 04:55:31.115507 3149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:31.115595 3149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:31.115696 3149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:31.115762 3149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:31.115857 3149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:31.125416 3149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:45259,127.31.93.61:34263,127.31.93.60:44579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.61.keytab
--principal=kudu/127.31.93.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.61:34263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.93.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:31.126649 3149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:31.127899 3149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:31.139183 3166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:31.139210 3169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:31.139490 3163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:31.140151 3149 server_base.cc:1034] running on GCE node
I20260809 04:55:31.140817 3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:31.142019 3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:31.143234 3149 hybrid_clock.cc:648] HybridClock initialized: now 1786251331143175 us; error 29 us; skew 500 ppm
I20260809 04:55:31.143518 3149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:31 dist-test-slave-4tdj krb5kdc[2998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251331, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:31.146507 3149 init.cc:377] Logged in from keytab as kudu/127.31.93.61@KRBTEST.COM (short username kudu)
I20260809 04:55:31.147769 3149 webserver.cc:460] Webserver started at http://127.31.93.61:34047/ using document root <none> and password file <none>
I20260809 04:55:31.148382 3149 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:31.148479 3149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:31.148766 3149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:31.151530 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/data/instance:
uuid: "961863a4cd424b1eb66139af1f8dd1b0"
format_stamp: "Formatted at 2026-08-09 04:55:31 on dist-test-slave-4tdj"
server_key: "bc64e719d1df0398670630b4a772cf07"
server_key_iv: "5563ef668be6d985e93a80ab3bff81c1"
server_key_version: "encryptionkey@0"
I20260809 04:55:31.152273 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/wal/instance:
uuid: "961863a4cd424b1eb66139af1f8dd1b0"
format_stamp: "Formatted at 2026-08-09 04:55:31 on dist-test-slave-4tdj"
server_key: "bc64e719d1df0398670630b4a772cf07"
server_key_iv: "5563ef668be6d985e93a80ab3bff81c1"
server_key_version: "encryptionkey@0"
I20260809 04:55:31.157013 3149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:55:31.160130 3176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:31.161280 3149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 04:55:31.161432 3149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/wal
uuid: "961863a4cd424b1eb66139af1f8dd1b0"
format_stamp: "Formatted at 2026-08-09 04:55:31 on dist-test-slave-4tdj"
server_key: "bc64e719d1df0398670630b4a772cf07"
server_key_iv: "5563ef668be6d985e93a80ab3bff81c1"
server_key_version: "encryptionkey@0"
I20260809 04:55:31.161613 3149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:31.178943 3149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:31.182257 3149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:31.182497 3149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:31.198223 3149 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.61:34263
I20260809 04:55:31.198246 3257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.61:34263 every 8 connection(s)
I20260809 04:55:31.199368 3149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/data/info.pb
I20260809 04:55:31.202646 3260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:31.207190 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 3149
I20260809 04:55:31.207398 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-1/wal/instance
I20260809 04:55:31.207948 32116 external_mini_cluster.cc:1428] Setting key 964ecd33fbf529b24d2c1a9e8d58e52d
I20260809 04:55:31.209405 3260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:55:31 dist-test-slave-4tdj krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251331, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.61@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:55:31.229110 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.212241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60788 (local address 127.31.93.62:45259)
0809 04:55:31.212424 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:31.212430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:31.213463 (+ 1033us) server_negotiation.cc:407] Connection header received
0809 04:55:31.215306 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:31.215312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:31.215387 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:31.215486 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:31.216790 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.217710 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.219167 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.219703 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.219829 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.223052 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:31.223059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:31.223063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:31.223102 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:31.226205 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.226892 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.226897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.226901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.227056 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.227428 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.227432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.227435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.227777 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:31.227906 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:31.228598 (+ 692us) server_negotiation.cc:299] Negotiation successful
0809 04:55:31.228808 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:55:31.229796 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.212601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45259 (local address 127.0.0.1:60788)
0809 04:55:31.213302 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.213346 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:31.215056 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:31.215661 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:31.215669 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:31.216131 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:31.216622 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:31.216638 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.217839 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.217842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:31.219023 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:31.219036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.219876 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.219880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:31.220059 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.220629 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:31.220651 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:31.222844 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:31.226351 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:31.226357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:31.226368 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:31.226768 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:31.227156 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:31.227159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:31.227161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:31.227319 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:31.227877 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:31.227882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:31.228196 (+ 314us) client_negotiation.cc:769] Sending connection context
0809 04:55:31.228431 (+ 235us) client_negotiation.cc:240] Negotiation successful
0809 04:55:31.228784 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":280,"threads_started":1}
I20260809 04:55:31.231418 3260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.93.60@KRBTEST.COM; defaulting to no policy
Aug 09 04:55:31 dist-test-slave-4tdj krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251331, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.61@KRBTEST.COM for kudu/127.31.93.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.60@KRBTEST.COM" created.
I20260809 04:55:31.245628 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.232151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:34263 (local address 127.0.0.1:36810)
0809 04:55:31.232344 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.232361 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:31.232456 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:31.233407 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:31.233412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:31.233828 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:31.233992 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:31.234002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.235348 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.235353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:31.236649 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:31.236665 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.237441 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.237448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:31.237639 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.238154 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:31.238176 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:31.240356 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:31.243430 (+ 3074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:31.243433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:31.243436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:31.243771 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:31.244125 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:31.244128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:31.244130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:31.244247 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:31.244846 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:31.244849 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:31.244967 (+ 118us) client_negotiation.cc:769] Sending connection context
0809 04:55:31.245094 (+ 127us) client_negotiation.cc:240] Negotiation successful
0809 04:55:31.245340 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 04:55:31.245846 3264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.232402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36810 (local address 127.31.93.61:34263)
0809 04:55:31.233000 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:31.233005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:31.233029 (+ 24us) server_negotiation.cc:407] Connection header received
0809 04:55:31.233069 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:31.233074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:31.233239 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:31.233342 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:31.234150 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.235219 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.236782 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.237298 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.237386 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.240531 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:31.240542 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:31.240553 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:31.240584 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:31.243309 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.243892 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.243901 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.243904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.244038 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.244340 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.244344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.244347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.244729 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:31.244920 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:31.245235 (+ 315us) server_negotiation.cc:299] Negotiation successful
0809 04:55:31.245454 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":326,"threads_started":1}
I20260809 04:55:31.247912 3260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:55:31.248955 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.248558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44040)
0809 04:55:31.248722 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.248824 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 04:55:31.249194 3180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
I20260809 04:55:31.250661 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.250365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44052)
0809 04:55:31.250491 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.250571 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260809 04:55:31.250844 3260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:44579: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.60.keytab.
Entry for principal kudu/127.31.93.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.60.keytab.
I20260809 04:55:31.274741 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.60:44579
--webserver_interface=127.31.93.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.60.keytab
--principal=kudu/127.31.93.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:45259,127.31.93.61:34263,127.31.93.60:44579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:55:31.300110 3260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } attempt: 1
I20260809 04:55:31.301143 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.300712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44058)
0809 04:55:31.300895 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.300996 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 04:55:31.302119 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.301821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44064)
0809 04:55:31.301929 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.302014 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260809 04:55:31.302340 3260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:44579: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
I20260809 04:55:31.370539 3260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } attempt: 2
I20260809 04:55:31.371629 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.371162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44074)
0809 04:55:31.371352 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.371458 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260809 04:55:31.372568 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.372234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44086)
0809 04:55:31.372373 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.372443 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 04:55:31.372764 3260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:44579: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
I20260809 04:55:31.411099 3134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } attempt: 5
Aug 09 04:55:31 dist-test-slave-4tdj krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251330, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for kudu/127.31.93.61@KRBTEST.COM
I20260809 04:55:31.423640 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.411735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:34263 (local address 127.0.0.1:36820)
0809 04:55:31.411913 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.411934 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:31.412065 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:31.412415 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:31.412420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:31.412819 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:31.413008 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:31.413018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.414001 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.414006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:31.414915 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:31.414929 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.415653 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.415659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:31.415848 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.416410 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:31.416429 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:31.418633 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:31.421360 (+ 2727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:31.421366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:31.421370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:31.421729 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:31.422059 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:31.422073 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:31.422077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:31.422213 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:31.422732 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:31.422737 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:31.422884 (+ 147us) client_negotiation.cc:769] Sending connection context
0809 04:55:31.423005 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 04:55:31.423281 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 04:55:31.423635 3264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.411862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.31.93.61:34263)
0809 04:55:31.412042 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:31.412047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:31.412083 (+ 36us) server_negotiation.cc:407] Connection header received
0809 04:55:31.412178 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:31.412184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:31.412271 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:31.412352 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:31.413158 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.413846 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.415076 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.415507 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.415617 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.418804 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:31.418809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:31.418812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:31.418845 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:31.421237 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.421836 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.421841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.421843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.421957 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.422334 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.422339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.422342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.422619 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:31.422750 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:31.423118 (+ 368us) server_negotiation.cc:299] Negotiation successful
0809 04:55:31.423313 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 04:55:31.424722 3134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:55:31.425789 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.425435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44096)
0809 04:55:31.425591 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.425663 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 04:55:31.426748 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.426449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44110)
0809 04:55:31.426574 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.426652 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 04:55:31.426941 3134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:44579: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
I20260809 04:55:31.463958 3260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } attempt: 3
I20260809 04:55:31.465116 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.464637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44122)
0809 04:55:31.464852 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.464957 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 04:55:31.466122 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.465727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44128)
0809 04:55:31.465944 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.466017 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260809 04:55:31.466332 3260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:44579: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
I20260809 04:55:31.478140 3134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } attempt: 1
I20260809 04:55:31.479139 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.478731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44138)
0809 04:55:31.478891 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.478971 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 04:55:31.480013 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.479706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44142)
0809 04:55:31.479845 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.479911 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 04:55:31.480203 3134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:44579: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
I20260809 04:55:31.548388 3134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } attempt: 2
I20260809 04:55:31.549445 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.548980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44154)
0809 04:55:31.549198 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.549284 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 04:55:31.550417 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.550042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44166)
0809 04:55:31.550206 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.550274 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 04:55:31.550603 3134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:44579: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
W20260809 04:55:31.590991 3270 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 04:55:31.591301 3270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:31.591392 3270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:31.591503 3270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:31.598336 3270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:31.598419 3270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:31.598503 3270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:31.598564 3270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:31.598645 3270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:31.607769 3270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:45259,127.31.93.61:34263,127.31.93.60:44579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.60.keytab
--principal=kudu/127.31.93.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.60:44579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.93.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:31.608915 3270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:31.609515 3260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } attempt: 4
I20260809 04:55:31.610157 3270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 04:55:31.610594 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.610100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44178)
0809 04:55:31.610353 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.610439 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 04:55:31.611519 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.611205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44190)
0809 04:55:31.611350 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.611417 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 04:55:31.611724 3260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:44579: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
W20260809 04:55:31.620725 3277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:31.620803 3278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:31.620944 3283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:31.621836 3270 server_base.cc:1034] running on GCE node
I20260809 04:55:31.622573 3270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:31.623780 3270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:31.624995 3270 hybrid_clock.cc:648] HybridClock initialized: now 1786251331624929 us; error 35 us; skew 500 ppm
I20260809 04:55:31.625296 3270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:31 dist-test-slave-4tdj krb5kdc[2998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251331, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:31.628156 3270 init.cc:377] Logged in from keytab as kudu/127.31.93.60@KRBTEST.COM (short username kudu)
I20260809 04:55:31.629460 3270 webserver.cc:460] Webserver started at http://127.31.93.60:33477/ using document root <none> and password file <none>
I20260809 04:55:31.630061 3270 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:31.630142 3270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:31.630445 3270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:31.633129 3270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/data/instance:
uuid: "ebf547ebd6094d299fd365b34cf622fa"
format_stamp: "Formatted at 2026-08-09 04:55:31 on dist-test-slave-4tdj"
server_key: "5eca2a3a14b2f1767d676af9ca1e5822"
server_key_iv: "1e416728ff4ddd64ac06a172cd5528d4"
server_key_version: "encryptionkey@0"
I20260809 04:55:31.633839 3270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/wal/instance:
uuid: "ebf547ebd6094d299fd365b34cf622fa"
format_stamp: "Formatted at 2026-08-09 04:55:31 on dist-test-slave-4tdj"
server_key: "5eca2a3a14b2f1767d676af9ca1e5822"
server_key_iv: "1e416728ff4ddd64ac06a172cd5528d4"
server_key_version: "encryptionkey@0"
I20260809 04:55:31.638441 3270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.006s
I20260809 04:55:31.641477 3300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:31.642692 3270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 04:55:31.642844 3270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/wal
uuid: "ebf547ebd6094d299fd365b34cf622fa"
format_stamp: "Formatted at 2026-08-09 04:55:31 on dist-test-slave-4tdj"
server_key: "5eca2a3a14b2f1767d676af9ca1e5822"
server_key_iv: "1e416728ff4ddd64ac06a172cd5528d4"
server_key_version: "encryptionkey@0"
I20260809 04:55:31.643023 3270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:31.656792 3134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } attempt: 3
I20260809 04:55:31.657718 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.657286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44194)
0809 04:55:31.657488 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.657569 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 04:55:31.658691 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.658316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44198)
0809 04:55:31.658517 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.658590 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260809 04:55:31.658885 3134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:44579: Network error: Client connection negotiation failed: client connection to 127.31.93.60:44579: connect: Connection refused (error 111)
I20260809 04:55:31.662725 3270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:31.666229 3270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:31.666465 3270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:31.682947 3270 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.60:44579
I20260809 04:55:31.682965 3395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.60:44579 every 8 connection(s)
I20260809 04:55:31.684119 3270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/data/info.pb
I20260809 04:55:31.687328 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 3270
I20260809 04:55:31.687496 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786250977734564-32116-0/minicluster-data/master-2/wal/instance
I20260809 04:55:31.687664 3396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:31.687989 32116 external_mini_cluster.cc:1428] Setting key 74e000103e98db5c574d40d3e0347208
I20260809 04:55:31.696457 3396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:55:31 dist-test-slave-4tdj krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.60@KRBTEST.COM
Aug 09 04:55:31 dist-test-slave-4tdj krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251331, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.60@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:55:31.708673 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.689571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44200 (local address 127.31.93.60:44579)
0809 04:55:31.690390 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:31.690405 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:31.690459 (+ 54us) server_negotiation.cc:407] Connection header received
0809 04:55:31.692339 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:31.692365 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:31.692825 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:31.693238 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:31.694545 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.695755 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.696973 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.697646 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.697805 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.702069 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:31.702076 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:31.702085 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:31.702123 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:31.704943 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.705618 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.705626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.705634 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.705855 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.706187 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.706192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.706195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.706704 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:31.706927 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:31.707310 (+ 383us) server_negotiation.cc:299] Negotiation successful
0809 04:55:31.707579 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":355,"threads_started":1}
I20260809 04:55:31.712958 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.698629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:45259 (local address 127.0.0.1:60796)
0809 04:55:31.699298 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.699337 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:31.699481 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:31.699817 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:31.699823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:31.700386 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:31.700609 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:31.700618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.701573 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.701577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:31.702643 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:31.702655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.703373 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.703379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:31.703511 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.704001 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:31.704022 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:31.706186 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:31.709758 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:31.709772 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:31.709779 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:31.710283 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:31.710744 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:31.710748 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:31.710752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:31.710874 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:31.711904 (+ 1030us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:31.711911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:31.712086 (+ 175us) client_negotiation.cc:769] Sending connection context
0809 04:55:31.712284 (+ 198us) client_negotiation.cc:240] Negotiation successful
0809 04:55:31.712571 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":338,"threads_started":1}
I20260809 04:55:31.712965 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.698628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.31.93.62:45259)
0809 04:55:31.698795 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:31.698799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:31.699423 (+ 624us) server_negotiation.cc:407] Connection header received
0809 04:55:31.699599 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:31.699603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:31.699663 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:31.699742 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:31.700747 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.701430 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.702771 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.703173 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.703280 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.706388 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:31.706393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:31.706396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:31.706429 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:31.709596 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.710425 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.710430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.710433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.710606 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.710978 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.710984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.710988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.711766 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:31.711897 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:31.712386 (+ 489us) server_negotiation.cc:299] Negotiation successful
0809 04:55:31.712602 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 04:55:31.714390 3396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:55:31 dist-test-slave-4tdj krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
Aug 09 04:55:31 dist-test-slave-4tdj krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251331, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.60@KRBTEST.COM for kudu/127.31.93.61@KRBTEST.COM
I20260809 04:55:31.726531 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.711236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.31.93.62:45259)
0809 04:55:31.711770 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:31.711775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:31.711797 (+ 22us) server_negotiation.cc:407] Connection header received
0809 04:55:31.711842 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:31.711846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:31.711906 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:31.712000 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:31.713989 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.715067 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.716653 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.717072 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.717173 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.721337 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:31.721344 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:31.721349 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:31.721387 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:31.724027 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.724745 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.724749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.724751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.724855 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.725205 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.725208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.725210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.725460 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:31.725724 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:31.725972 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 04:55:31.726170 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":235,"threads_started":1}
I20260809 04:55:31.727792 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.715147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:34263 (local address 127.0.0.1:36824)
0809 04:55:31.715378 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.715394 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:31.715491 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:31.715823 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:31.715828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:31.716215 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:31.716377 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:31.716388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.717667 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.717672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:31.718676 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:31.718688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.719592 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.719596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:31.719707 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.720632 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:31.720648 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:31.722507 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:31.725549 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:31.725554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:31.725558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:31.725883 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:31.726243 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:31.726248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:31.726252 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:31.726443 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:31.726959 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:31.726963 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:31.727111 (+ 148us) client_negotiation.cc:769] Sending connection context
0809 04:55:31.727215 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 04:55:31.727454 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260809 04:55:31.728467 3264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.715134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.31.93.61:34263)
0809 04:55:31.715296 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:31.715301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:31.715457 (+ 156us) server_negotiation.cc:407] Connection header received
0809 04:55:31.715611 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:31.715616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:31.715692 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:31.715793 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:31.716526 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.717521 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.718813 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.719464 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.719597 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.722675 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:31.722680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:31.722683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:31.722715 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:31.725439 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.725989 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.725995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.725998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.726140 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.726535 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.726540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.726543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.726836 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:31.726974 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:31.727336 (+ 362us) server_negotiation.cc:299] Negotiation successful
0809 04:55:31.727546 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 04:55:31.729385 3396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:55:31 dist-test-slave-4tdj krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251331, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.60@KRBTEST.COM for kudu/127.31.93.60@KRBTEST.COM
Aug 09 04:55:31 dist-test-slave-4tdj krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.61@KRBTEST.COM
I20260809 04:55:31.741763 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.730010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44208)
0809 04:55:31.730247 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.730264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:31.730392 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:31.730652 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:31.730657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:31.731080 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:31.731234 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:31.731245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.732487 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.732491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:31.733643 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:31.733656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.734497 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.734503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:31.734710 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.735204 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:31.735223 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:31.736986 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:31.739698 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:31.739703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:31.739707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:31.740039 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:31.740340 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:31.740345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:31.740349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:31.740468 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:31.740901 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:31.740905 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:31.741038 (+ 133us) client_negotiation.cc:769] Sending connection context
0809 04:55:31.741154 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 04:55:31.741419 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260809 04:55:31.741921 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.729995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.31.93.60:44579)
0809 04:55:31.730136 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:31.730140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:31.730334 (+ 194us) server_negotiation.cc:407] Connection header received
0809 04:55:31.730473 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:31.730477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:31.730542 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:31.730610 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:31.731363 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.732375 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.733776 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.734345 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.734470 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.737150 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:31.737155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:31.737160 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:31.737194 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:31.739583 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.740134 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.740139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.740141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.740238 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.740561 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.740566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.740569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.740798 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:31.740915 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:31.741252 (+ 337us) server_negotiation.cc:299] Negotiation successful
0809 04:55:31.741589 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":46}
I20260809 04:55:31.742887 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.727944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.31.93.61:34263)
0809 04:55:31.728570 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:31.728576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:31.728599 (+ 23us) server_negotiation.cc:407] Connection header received
0809 04:55:31.728643 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:31.728647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:31.728715 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:31.728822 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:31.730148 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.731268 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.732804 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.733413 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.733538 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.737885 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:31.737893 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:31.737897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:31.737932 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:31.740252 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.740865 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.740869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.740871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.740985 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.741336 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.741340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.741344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.741580 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:31.741825 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:31.742315 (+ 490us) server_negotiation.cc:299] Negotiation successful
0809 04:55:31.742517 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":352,"threads_started":1}
I20260809 04:55:31.746906 3396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa: Bootstrap starting.
I20260809 04:55:31.747268 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:31.750749 3396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:31.751721 3396 log.cc:824] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:31.750749 3084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0a61f438e7094124afa4a3fab2c2a07f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60812
I20260809 04:55:31.754267 3396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa: No bootstrap required, opened a new log
I20260809 04:55:31.760859 3396 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } }
I20260809 04:55:31.761138 3396 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf547ebd6094d299fd365b34cf622fa, State: Initialized, Role: FOLLOWER
I20260809 04:55:31.761543 3396 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } }
I20260809 04:55:31.762336 3412 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } } }
I20260809 04:55:31.762638 3412 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:55:31.763329 3396 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa [sys.catalog]: configured and running, proceeding with master startup.
W20260809 04:55:31.772842 3430 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:55:31.772966 3430 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:55:31.770864 3270 master_runner.cc:186] Error getting master registration for 127.31.93.62:45259: OK, instance_id { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" instance_seqno: 1786251330704316 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 04:55:31.781636 3270 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 04:55:31.836083 3134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } attempt: 4
Aug 09 04:55:31 dist-test-slave-4tdj krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251330, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for kudu/127.31.93.60@KRBTEST.COM
I20260809 04:55:31.852752 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.836847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44224 (local address 127.31.93.60:44579)
0809 04:55:31.837267 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:31.837271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:31.837285 (+ 14us) server_negotiation.cc:407] Connection header received
0809 04:55:31.837334 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:31.837337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:31.837396 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:31.837464 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:31.838528 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.839696 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.841469 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.842171 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.842281 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.846255 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:31.846262 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:31.846267 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:31.846329 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:31.849756 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.850591 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.850597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.850601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.850752 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.851191 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.851196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.851199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.851552 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:31.851711 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:31.852174 (+ 463us) server_negotiation.cc:299] Negotiation successful
0809 04:55:31.852375 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295}
I20260809 04:55:31.852814 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.836763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44224)
0809 04:55:31.836945 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.836965 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:31.837073 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:31.837568 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:31.837574 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:31.838106 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:31.838361 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:31.838376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.839852 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.839859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:31.841327 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:31.841345 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.842346 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.842353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:31.842579 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.843239 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:31.843262 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:31.846030 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:31.849896 (+ 3866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:31.849903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:31.849907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:31.850475 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:31.850866 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:31.850872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:31.850875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:31.851074 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:31.851691 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:31.851697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:31.851885 (+ 188us) client_negotiation.cc:769] Sending connection context
0809 04:55:31.852057 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 04:55:31.852379 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 04:55:31.858744 3134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f: Bootstrap starting.
I20260809 04:55:31.861876 3134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:31.862942 3134 log.cc:824] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:31.866122 3134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f: No bootstrap required, opened a new log
I20260809 04:55:31.873397 3134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } }
I20260809 04:55:31.873790 3134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a61f438e7094124afa4a3fab2c2a07f, State: Initialized, Role: FOLLOWER
I20260809 04:55:31.874310 3134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } }
I20260809 04:55:31.875654 3436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } } }
I20260809 04:55:31.876191 3436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:55:31.876647 3134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:31.876871 3084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 04:55:31.877103 3084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:31.880482 3084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } }
I20260809 04:55:31.882921 3356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a61f438e7094124afa4a3fab2c2a07f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ebf547ebd6094d299fd365b34cf622fa"
I20260809 04:55:31.883453 3356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:31.887059 3356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a61f438e7094124afa4a3fab2c2a07f in term 1.
I20260809 04:55:31.887462 3084 leader_election.cc:288] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [CANDIDATE]: Term 1 election: Requested vote from peers 961863a4cd424b1eb66139af1f8dd1b0 (127.31.93.61:34263), ebf547ebd6094d299fd365b34cf622fa (127.31.93.60:44579)
I20260809 04:55:31.887815 3048 leader_election.cc:302] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a61f438e7094124afa4a3fab2c2a07f, ebf547ebd6094d299fd365b34cf622fa; no voters:
I20260809 04:55:31.888307 3436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:31.889030 3436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [term 1 LEADER]: Becoming Leader. State: Replica: 0a61f438e7094124afa4a3fab2c2a07f, State: Running, Role: LEADER
I20260809 04:55:31.889598 3436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } }
I20260809 04:55:31.901604 3446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a61f438e7094124afa4a3fab2c2a07f. Latest consensus state: current_term: 1 leader_uuid: "0a61f438e7094124afa4a3fab2c2a07f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } } }
I20260809 04:55:31.902089 3446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:31.910918 3260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } attempt: 5
W20260809 04:55:31.918256 3313 tablet.cc:2369] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:55:31.906594 3019 master_runner.cc:186] Error getting master registration for 127.31.93.61:34263: OK, instance_id { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" instance_seqno: 1786251331193688 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
Aug 09 04:55:31 dist-test-slave-4tdj krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251331, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.61@KRBTEST.COM for kudu/127.31.93.60@KRBTEST.COM
I20260809 04:55:31.919907 3019 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 04:55:31.929919 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.911503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:44579 (local address 127.0.0.1:44238)
0809 04:55:31.911695 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:31.911719 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:31.911842 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:31.912209 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:31.912215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:31.912737 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:31.912963 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:31.912989 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.914474 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.914480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:31.915983 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:31.915998 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.917236 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.917241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:31.917432 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.918030 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:31.918053 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:31.920647 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:31.927115 (+ 6468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:31.927120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:31.927124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:31.927626 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:31.928090 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:31.928096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:31.928101 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:31.928270 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:31.928931 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:31.928937 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:31.929115 (+ 178us) client_negotiation.cc:769] Sending connection context
0809 04:55:31.929251 (+ 136us) client_negotiation.cc:240] Negotiation successful
0809 04:55:31.929534 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 04:55:31.931192 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.911598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.31.93.60:44579)
0809 04:55:31.911772 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:31.911778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:31.911797 (+ 19us) server_negotiation.cc:407] Connection header received
0809 04:55:31.911971 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:31.911977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:31.912070 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:31.912175 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:31.913130 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.914240 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.916210 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.916885 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.916996 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.920821 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:31.920827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:31.920831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:31.920860 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:31.926877 (+ 6017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.927786 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.927790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.927794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.927949 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.928422 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.928428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.928432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.928792 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:31.930530 (+ 1738us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:31.930653 (+ 123us) server_negotiation.cc:299] Negotiation successful
0809 04:55:31.930868 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260809 04:55:31.931679 3457 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:55:31.931814 3457 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:55:31.935433 3459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:31.937747 3459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260809 04:55:31.944756 3052 tablet.cc:2369] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:55:31.945084 3260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0: Bootstrap starting.
I20260809 04:55:31.949880 3260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:31.955494 3260 log.cc:824] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:31.972321 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.924229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.31.93.60:44579)
0809 04:55:31.928032 (+ 3803us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:31.928037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:31.942935 (+ 14898us) server_negotiation.cc:407] Connection header received
0809 04:55:31.943132 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:31.943140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:31.943273 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:31.943425 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:31.947510 (+ 4085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.949501 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.952866 (+ 3365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.954032 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.954216 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.959975 (+ 5759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:31.959993 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:31.959999 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:31.960066 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:31.966083 (+ 6017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.967582 (+ 1499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.967588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.967592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.967808 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.968566 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.968572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.968575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.969242 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:31.969634 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:31.970659 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0809 04:55:31.971052 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3253,"thread_start_us":2705,"threads_started":1}
I20260809 04:55:31.969514 3260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0: No bootstrap required, opened a new log
I20260809 04:55:31.977759 3356 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa [term 1 FOLLOWER]: Refusing update from remote peer 0a61f438e7094124afa4a3fab2c2a07f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:55:31.978940 3446 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:55:31.994876 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.931791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.31.93.61:34263)
0809 04:55:31.931956 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:31.931962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:31.973676 (+ 41714us) server_negotiation.cc:407] Connection header received
0809 04:55:31.974040 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:31.974046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:31.974139 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:31.974209 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:31.976542 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.977601 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.980367 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.981067 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.981166 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.986616 (+ 5450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:31.986623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:31.986628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:31.986668 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:31.990429 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.991756 (+ 1327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.991762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.991765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.991929 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:31.992741 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:31.992747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:31.992751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:31.993107 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:31.993338 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:31.994259 (+ 921us) server_negotiation.cc:299] Negotiation successful
0809 04:55:31.994538 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 04:55:32.007210 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:31.923347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.31.93.62:45259)
0809 04:55:31.923552 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:31.923558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:31.930613 (+ 7055us) server_negotiation.cc:407] Connection header received
0809 04:55:31.931010 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:31.931016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:31.931132 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:31.931201 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:31.956657 (+ 25456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.957865 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.964415 (+ 6550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:31.965352 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:31.965486 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:31.989851 (+ 24365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:31.989860 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:31.989867 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:31.989919 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:31.994239 (+ 4320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:32.002988 (+ 8749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:32.002996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:32.003001 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:32.003187 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:32.004500 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:32.004505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:32.004509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:32.004915 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:32.005208 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:32.005982 (+ 774us) server_negotiation.cc:299] Negotiation successful
0809 04:55:32.006467 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:55:32.011499 3412 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a61f438e7094124afa4a3fab2c2a07f. Latest consensus state: current_term: 1 leader_uuid: "0a61f438e7094124afa4a3fab2c2a07f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } } }
I20260809 04:55:32.011881 3412 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:55:32.018762 3469 mvcc.cc:205] Tried to move back new op lower bound from 7316485455750520832 to 7316485455425204224. Current Snapshot: MvccSnapshot[applied={T|T < 7316485455750520832}]
I20260809 04:55:32.020530 3446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a61f438e7094124afa4a3fab2c2a07f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } } }
I20260809 04:55:32.020824 3446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:32.021181 3459 catalog_manager.cc:1347] Generated new cluster ID: 7fa6403d5e534c77836c65c1d64b7c3d
I20260809 04:55:32.021284 3459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:32.029330 3471 mvcc.cc:205] Tried to move back new op lower bound from 7316485455750520832 to 7316485455425204224. Current Snapshot: MvccSnapshot[applied={T|T < 7316485455750520832}]
I20260809 04:55:32.046885 3412 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a61f438e7094124afa4a3fab2c2a07f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } } }
I20260809 04:55:32.047170 3412 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:55:32.051043 3459 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:32.052906 3459 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:32.054952 3260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } }
I20260809 04:55:32.055438 3260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 961863a4cd424b1eb66139af1f8dd1b0, State: Initialized, Role: FOLLOWER
I20260809 04:55:32.058799 3260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } }
I20260809 04:55:32.060130 3477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } } }
I20260809 04:55:32.060881 3477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:55:32.065403 3260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:32.066820 3216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:32.070901 3218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a61f438e7094124afa4a3fab2c2a07f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "961863a4cd424b1eb66139af1f8dd1b0"
I20260809 04:55:32.075174 3459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f: Generated new TSK 0
I20260809 04:55:32.075448 3216 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0 [term 1 FOLLOWER]: Refusing update from remote peer 0a61f438e7094124afa4a3fab2c2a07f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:55:32.076259 3459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:32.079180 3218 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a61f438e7094124afa4a3fab2c2a07f in term 1.
I20260809 04:55:32.080862 3446 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [LEADER]: Connected to new peer: Peer: permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:55:32.128211 3477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a61f438e7094124afa4a3fab2c2a07f. Latest consensus state: current_term: 1 leader_uuid: "0a61f438e7094124afa4a3fab2c2a07f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } } }
I20260809 04:55:32.128615 3477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:55:32.129009 3488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a61f438e7094124afa4a3fab2c2a07f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } } }
I20260809 04:55:32.129451 3488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 04:55:32.198109 32116 mini_kdc.cc:348] Time spent kdestroy: real 0.109s user 0.000s sys 0.028s
I20260809 04:55:32.246068 3503 catalog_manager.cc:1259] Loaded cluster ID: 7fa6403d5e534c77836c65c1d64b7c3d
I20260809 04:55:32.246258 3503 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0: loading cluster ID for follower catalog manager: success
I20260809 04:55:32.250600 3503 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0: acquiring CA information for follower catalog manager: success
I20260809 04:55:32.253417 3503 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 961863a4cd424b1eb66139af1f8dd1b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 04:55:32.271548 3406 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:32.245083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.31.93.62:45259)
0809 04:55:32.245299 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.245304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.268150 (+ 22846us) server_negotiation.cc:407] Connection header received
0809 04:55:32.268398 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.268404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.268492 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.268581 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:32.271281 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60834: BlockingRecv error: recv got EOF from 127.0.0.1:60834 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260809 04:55:32.273615 3407 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:32.252682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.31.93.61:34263)
0809 04:55:32.252875 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.252882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.271626 (+ 18744us) server_negotiation.cc:407] Connection header received
0809 04:55:32.272127 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.272133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.272217 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.272296 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:32.273294 (+ 998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36856: BlockingRecv error: recv got EOF from 127.0.0.1:36856 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260809 04:55:32.276086 3466 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:32.253036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44254 (local address 127.31.93.60:44579)
0809 04:55:32.253178 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.253183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.274274 (+ 21091us) server_negotiation.cc:407] Connection header received
0809 04:55:32.274636 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.274647 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.274758 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.274842 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:32.275838 (+ 996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44254: BlockingRecv error: recv got EOF from 127.0.0.1:44254 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 04:55:32.327085 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:32.311721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36860 (local address 127.31.93.61:34263)
0809 04:55:32.311907 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.311913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.318728 (+ 6815us) server_negotiation.cc:407] Connection header received
0809 04:55:32.319704 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.319712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.319932 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.320065 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:32.320911 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.322053 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.324029 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.324725 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.324823 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:32.325563 (+ 740us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:32.326229 (+ 666us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:32.326375 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:32.326628 (+ 253us) server_negotiation.cc:299] Negotiation successful
0809 04:55:32.326720 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260809 04:55:32.372645 3466 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:32.362822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.31.93.60:44579)
0809 04:55:32.363006 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.363012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.370386 (+ 7374us) server_negotiation.cc:407] Connection header received
0809 04:55:32.370982 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.370989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.371115 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.371190 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:32.372388 (+ 1198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44262: BlockingRecv error: recv got EOF from 127.0.0.1:44262 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 04:55:32.409163 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:32.395731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.31.93.61:34263)
0809 04:55:32.395902 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.395908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.402254 (+ 6346us) server_negotiation.cc:407] Connection header received
0809 04:55:32.402582 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.402589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.402694 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.402779 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:32.403578 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.404742 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.406836 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.407621 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.407778 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:32.408343 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:32.408494 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:32.408588 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:32.408754 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 04:55:32.408838 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260809 04:55:32.421947 3466 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:32.419764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44274 (local address 127.31.93.60:44579)
0809 04:55:32.419933 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.419939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.420456 (+ 517us) server_negotiation.cc:407] Connection header received
0809 04:55:32.420644 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.420651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.420753 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.420879 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:32.421697 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44274: BlockingRecv error: recv got EOF from 127.0.0.1:44274 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:55:32.436048 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:32.429084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.31.93.61:34263)
0809 04:55:32.429264 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.429269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.429796 (+ 527us) server_negotiation.cc:407] Connection header received
0809 04:55:32.429955 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.429961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.430066 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.430179 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:32.430866 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.432004 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.433933 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.434661 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.434811 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:32.435284 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:32.435427 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:32.435523 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:32.435671 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 04:55:32.435752 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260809 04:55:32.450477 3466 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:32.448301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44280 (local address 127.31.93.60:44579)
0809 04:55:32.448479 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.448496 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.449013 (+ 517us) server_negotiation.cc:407] Connection header received
0809 04:55:32.449202 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.449208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.449306 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.449415 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:32.450206 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44280: BlockingRecv error: recv got EOF from 127.0.0.1:44280 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:55:32.467135 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:32.460079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36880 (local address 127.31.93.61:34263)
0809 04:55:32.460308 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.460312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.460796 (+ 484us) server_negotiation.cc:407] Connection header received
0809 04:55:32.460968 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.460974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.461072 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.461181 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:32.461618 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.462794 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.464729 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.465521 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.465646 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:32.466232 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:32.466510 (+ 278us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:32.466599 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:32.466740 (+ 141us) server_negotiation.cc:299] Negotiation successful
0809 04:55:32.466856 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
W20260809 04:55:32.479219 3466 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:32.474427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.31.93.60:44579)
0809 04:55:32.474678 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.474683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.477632 (+ 2949us) server_negotiation.cc:407] Connection header received
0809 04:55:32.477775 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.477780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.477883 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.477998 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:32.478980 (+ 982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44286: BlockingRecv error: recv got EOF from 127.0.0.1:44286 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
I20260809 04:55:32.497084 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:32.490532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.31.93.61:34263)
0809 04:55:32.490695 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.490701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.491180 (+ 479us) server_negotiation.cc:407] Connection header received
0809 04:55:32.491320 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.491326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.491429 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.491523 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:32.491946 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.493061 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.495026 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.495751 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.495876 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:32.496310 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:32.496438 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:32.496526 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:32.496658 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 04:55:32.496749 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260809 04:55:32.511498 3466 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:32.504104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44300 (local address 127.31.93.60:44579)
0809 04:55:32.504277 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.504282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.510072 (+ 5790us) server_negotiation.cc:407] Connection header received
0809 04:55:32.510244 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.510249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.510364 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.510465 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:32.511266 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44300: BlockingRecv error: recv got EOF from 127.0.0.1:44300 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 04:55:32.536121 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:32.530461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36906 (local address 127.31.93.61:34263)
0809 04:55:32.530661 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.530666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.530934 (+ 268us) server_negotiation.cc:407] Connection header received
0809 04:55:32.531115 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.531119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.531183 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.531263 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:32.531718 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.532467 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.534429 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.534918 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.535023 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:32.535480 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:32.535583 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:32.535649 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:32.535799 (+ 150us) server_negotiation.cc:299] Negotiation successful
0809 04:55:32.535860 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260809 04:55:32.543517 3466 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:32.541712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.31.93.60:44579)
0809 04:55:32.541908 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.541913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.542223 (+ 310us) server_negotiation.cc:407] Connection header received
0809 04:55:32.542394 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.542398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.542466 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.542549 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:32.543317 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44312: BlockingRecv error: recv got EOF from 127.0.0.1:44312 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 04:55:32.585500 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:32.579864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36912 (local address 127.31.93.61:34263)
0809 04:55:32.580055 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.580060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.580391 (+ 331us) server_negotiation.cc:407] Connection header received
0809 04:55:32.580594 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.580598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.580668 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.580761 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:32.581232 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.582021 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.583877 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.584281 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.584375 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:32.584846 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:32.584959 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:32.585018 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:32.585175 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 04:55:32.585244 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260809 04:55:32.593276 3466 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:32.591463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44318 (local address 127.31.93.60:44579)
0809 04:55:32.591638 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.591644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.591906 (+ 262us) server_negotiation.cc:407] Connection header received
0809 04:55:32.592081 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.592085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.592161 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.592240 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:32.593038 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44318: BlockingRecv error: recv got EOF from 127.0.0.1:44318 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 04:55:32.622838 3446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a61f438e7094124afa4a3fab2c2a07f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a61f438e7094124afa4a3fab2c2a07f" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 45259 } } peers { permanent_uuid: "961863a4cd424b1eb66139af1f8dd1b0" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 34263 } } peers { permanent_uuid: "ebf547ebd6094d299fd365b34cf622fa" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 44579 } } }
I20260809 04:55:32.623160 3446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a61f438e7094124afa4a3fab2c2a07f [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:32.667670 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:32.661270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.31.93.61:34263)
0809 04:55:32.661532 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.661539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.661894 (+ 355us) server_negotiation.cc:407] Connection header received
0809 04:55:32.662075 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.662081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.662168 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.662269 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:32.662726 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.663880 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.665389 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.666190 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.666352 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:32.666815 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:32.666989 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:32.667087 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:32.667245 (+ 158us) server_negotiation.cc:299] Negotiation successful
0809 04:55:32.667334 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260809 04:55:32.676856 3466 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:32.674956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44330 (local address 127.31.93.60:44579)
0809 04:55:32.675171 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.675177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.675461 (+ 284us) server_negotiation.cc:407] Connection header received
0809 04:55:32.675646 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.675650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.675726 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.675807 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:32.676638 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44330: BlockingRecv error: recv got EOF from 127.0.0.1:44330 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260809 04:55:32.774923 3430 catalog_manager.cc:1259] Loaded cluster ID: 7fa6403d5e534c77836c65c1d64b7c3d
I20260809 04:55:32.775027 3430 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa: loading cluster ID for follower catalog manager: success
I20260809 04:55:32.777454 3430 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa: acquiring CA information for follower catalog manager: success
I20260809 04:55:32.779170 3430 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ebf547ebd6094d299fd365b34cf622fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 04:55:32.813021 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:32.808204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36926 (local address 127.31.93.61:34263)
0809 04:55:32.808430 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.808437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.808687 (+ 250us) server_negotiation.cc:407] Connection header received
0809 04:55:32.808887 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.808891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.808959 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.809037 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:32.809447 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.810146 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.811464 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.811890 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.811999 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:32.812379 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:32.812477 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:32.812549 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:32.812700 (+ 151us) server_negotiation.cc:299] Negotiation successful
0809 04:55:32.812777 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 04:55:32.823210 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:32.818098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.31.93.60:44579)
0809 04:55:32.818330 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.818336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.818556 (+ 220us) server_negotiation.cc:407] Connection header received
0809 04:55:32.818724 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.818729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.818863 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.818960 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:32.819343 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.820038 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.821220 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.821599 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.821712 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:32.822160 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:32.822661 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:32.822798 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:32.822887 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 04:55:32.822961 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 04:55:32.836028 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:32.828918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.31.93.61:34263)
0809 04:55:32.829307 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.829313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.830390 (+ 1077us) server_negotiation.cc:407] Connection header received
0809 04:55:32.830630 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.830635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.830719 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.830831 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:32.831505 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.832245 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.834161 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.834755 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.834861 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:32.835303 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:32.835426 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:32.835485 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:32.835688 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 04:55:32.835760 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
I20260809 04:55:32.836364 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:32.829086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44348 (local address 127.31.93.60:44579)
0809 04:55:32.829251 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.829256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.830593 (+ 1337us) server_negotiation.cc:407] Connection header received
0809 04:55:32.830788 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.830794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.830868 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.830951 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:32.831729 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.832713 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.834475 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.835070 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.835191 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:32.835641 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:32.835777 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:32.835855 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:32.836032 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 04:55:32.836103 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 04:55:32.836975 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:32.828619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60836 (local address 127.31.93.62:45259)
0809 04:55:32.829670 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:32.829686 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:32.830334 (+ 648us) server_negotiation.cc:407] Connection header received
0809 04:55:32.830459 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:32.830464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:32.830677 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:32.831365 (+ 688us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:32.831527 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.832562 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.834540 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:32.835147 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:32.835249 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:32.835759 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:32.836194 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:32.836282 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:32.836660 (+ 378us) server_negotiation.cc:299] Negotiation successful
0809 04:55:32.836746 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":937,"thread_start_us":841,"threads_started":1}
I20260809 04:55:32.841195 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 3019
I20260809 04:55:32.842173 3118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:55:32.951308 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 3019
I20260809 04:55:32.977584 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 3149
I20260809 04:55:32.978698 3247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:55:33.086966 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 3149
I20260809 04:55:33.111924 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 3270
I20260809 04:55:33.112947 3385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:55:33.216828 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 3270
2026-08-09T04:55:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4431 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 04:55:33 dist-test-slave-4tdj krb5kdc[3702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 04:55:33 dist-test-slave-4tdj krb5kdc[3702](info): set up 2 sockets
Aug 09 04:55:33 dist-test-slave-4tdj krb5kdc[3702](info): commencing operation
krb5kdc: starting...
W20260809 04:55:34.437184 32116 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/test-user.keytab.
Aug 09 04:55:34 dist-test-slave-4tdj krb5kdc[3702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T04:55:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T04:55:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.93.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
Entry for principal kudu/127.31.93.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab.
I20260809 04:55:34.687191 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:41793
--webserver_interface=127.31.93.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:41793,127.31.93.61:33657,127.31.93.60:41763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 04:55:35.003242 3728 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 04:55:35.003557 3728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:35.003656 3728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:35.003763 3728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:35.010978 3728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:35.011065 3728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:35.011148 3728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:35.011205 3728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:35.011287 3728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:35.021677 3728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:41793,127.31.93.61:33657,127.31.93.60:41763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.62.keytab
--principal=kudu/127.31.93.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.62:41793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.93.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:35.023284 3728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:35.024878 3728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:35.035836 3738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:35.035890 3740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:35.036000 3737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:35.036896 3728 server_base.cc:1034] running on GCE node
I20260809 04:55:35.037618 3728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:35.038987 3728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:35.040236 3728 hybrid_clock.cc:648] HybridClock initialized: now 1786251335040158 us; error 62 us; skew 500 ppm
I20260809 04:55:35.040540 3728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:35 dist-test-slave-4tdj krb5kdc[3702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251335, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:35.046006 3728 init.cc:377] Logged in from keytab as kudu/127.31.93.62@KRBTEST.COM (short username kudu)
I20260809 04:55:35.047433 3728 webserver.cc:460] Webserver started at http://127.31.93.62:45113/ using document root <none> and password file <none>
I20260809 04:55:35.048084 3728 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:35.048184 3728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:35.048480 3728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:35.051468 3728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/data/instance:
uuid: "5a970f39d7f947fea1ffb0ec9d146a5b"
format_stamp: "Formatted at 2026-08-09 04:55:35 on dist-test-slave-4tdj"
server_key: "ca45a03d0d5c104981ac6c0208bb5c40"
server_key_iv: "acf0b56503c35c71d78652807b417548"
server_key_version: "encryptionkey@0"
I20260809 04:55:35.052223 3728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/wal/instance:
uuid: "5a970f39d7f947fea1ffb0ec9d146a5b"
format_stamp: "Formatted at 2026-08-09 04:55:35 on dist-test-slave-4tdj"
server_key: "ca45a03d0d5c104981ac6c0208bb5c40"
server_key_iv: "acf0b56503c35c71d78652807b417548"
server_key_version: "encryptionkey@0"
I20260809 04:55:35.057013 3728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260809 04:55:35.060314 3750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:35.061518 3728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:55:35.061677 3728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/wal
uuid: "5a970f39d7f947fea1ffb0ec9d146a5b"
format_stamp: "Formatted at 2026-08-09 04:55:35 on dist-test-slave-4tdj"
server_key: "ca45a03d0d5c104981ac6c0208bb5c40"
server_key_iv: "acf0b56503c35c71d78652807b417548"
server_key_version: "encryptionkey@0"
I20260809 04:55:35.061858 3728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:35.076507 3728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:35.080087 3728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:35.080319 3728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:35.096791 3728 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.62:41793
I20260809 04:55:35.096810 3827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.62:41793 every 8 connection(s)
I20260809 04:55:35.098029 3728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/data/info.pb
I20260809 04:55:35.100306 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 3728
I20260809 04:55:35.100466 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-0/wal/instance
I20260809 04:55:35.100899 32116 external_mini_cluster.cc:1428] Setting key e06f8a1727763a63ab8646282291766a
I20260809 04:55:35.101609 3829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:35.107877 3829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:55:35 dist-test-slave-4tdj krb5kdc[3702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251335, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.93.61@KRBTEST.COM; defaulting to no policy
I20260809 04:55:35.128326 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.110834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41793 (local address 127.0.0.1:33118)
0809 04:55:35.111875 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.111917 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:35.113536 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:35.114283 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:35.114313 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:35.114962 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:35.115287 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:35.115303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:35.116791 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:35.116795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:35.117822 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:35.117834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:35.118728 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:35.118735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:35.118945 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:35.119536 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:35.119556 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:35.121565 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:35.124457 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:35.124468 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:35.124479 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:35.124887 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:35.125272 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:35.125277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:35.125281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:35.125425 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:35.126090 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:35.126098 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:35.126295 (+ 197us) client_negotiation.cc:769] Sending connection context
0809 04:55:35.126561 (+ 266us) client_negotiation.cc:240] Negotiation successful
0809 04:55:35.127106 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":176,"thread_start_us":653,"threads_started":1}
I20260809 04:55:35.128345 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.110832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.31.93.62:41793)
0809 04:55:35.111592 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:35.111606 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:35.112022 (+ 416us) server_negotiation.cc:407] Connection header received
0809 04:55:35.113854 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:35.113859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:35.114071 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:35.114236 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:35.115475 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:35.116626 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:35.117947 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:35.118503 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:35.118661 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:35.121773 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:35.121780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:35.121790 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:35.121818 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:35.124346 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:35.124988 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:35.124996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:35.124999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:35.125132 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:35.125522 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:35.125526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:35.125528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:35.125962 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:35.126139 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:35.126717 (+ 578us) server_negotiation.cc:299] Negotiation successful
0809 04:55:35.126972 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":544,"thread_start_us":342,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.61@KRBTEST.COM" created.
I20260809 04:55:35.130998 3829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:55:35.131953 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.131545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:33657 (local address 127.0.0.1:54228)
0809 04:55:35.131707 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.131820 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 04:55:35.132172 3754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.93.61:33657: connect: Connection refused (error 111)
I20260809 04:55:35.133822 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.133404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:33657 (local address 127.0.0.1:54230)
0809 04:55:35.133578 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.133661 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 04:55:35.134052 3829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.61:33657: Network error: Client connection negotiation failed: client connection to 127.31.93.61:33657: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.61.keytab.
Entry for principal kudu/127.31.93.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.61.keytab.
I20260809 04:55:35.163004 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.61:33657
--webserver_interface=127.31.93.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.61.keytab
--principal=kudu/127.31.93.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:41793,127.31.93.61:33657,127.31.93.60:41763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:55:35.184033 3829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } attempt: 1
I20260809 04:55:35.185360 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.184777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:33657 (local address 127.0.0.1:54244)
0809 04:55:35.185065 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.185164 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260809 04:55:35.186623 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.186136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:33657 (local address 127.0.0.1:54256)
0809 04:55:35.186357 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.186452 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 04:55:35.186872 3829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.61:33657: Network error: Client connection negotiation failed: client connection to 127.31.93.61:33657: connect: Connection refused (error 111)
I20260809 04:55:35.255119 3829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } attempt: 2
I20260809 04:55:35.256373 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.255897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:33657 (local address 127.0.0.1:54272)
0809 04:55:35.256109 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.256225 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260809 04:55:35.257453 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.257062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:33657 (local address 127.0.0.1:54284)
0809 04:55:35.257211 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.257302 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 04:55:35.257668 3829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.61:33657: Network error: Client connection negotiation failed: client connection to 127.31.93.61:33657: connect: Connection refused (error 111)
I20260809 04:55:35.348903 3829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } attempt: 3
I20260809 04:55:35.349964 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.349524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:33657 (local address 127.0.0.1:54296)
0809 04:55:35.349724 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.349820 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 04:55:35.351007 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.350648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:33657 (local address 127.0.0.1:54298)
0809 04:55:35.350798 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.350867 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 04:55:35.351195 3829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.61:33657: Network error: Client connection negotiation failed: client connection to 127.31.93.61:33657: connect: Connection refused (error 111)
W20260809 04:55:35.490064 3845 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 04:55:35.490398 3845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:35.490497 3845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:35.490605 3845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 04:55:35.494378 3829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } attempt: 4
I20260809 04:55:35.495486 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.494970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:33657 (local address 127.0.0.1:54304)
0809 04:55:35.495266 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.495355 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260809 04:55:35.496403 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.496056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:33657 (local address 127.0.0.1:54314)
0809 04:55:35.496198 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.496282 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.61:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 04:55:35.496620 3829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.61:33657: Network error: Client connection negotiation failed: client connection to 127.31.93.61:33657: connect: Connection refused (error 111)
W20260809 04:55:35.497651 3845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:35.497735 3845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:35.497818 3845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:35.497880 3845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:35.497960 3845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:35.507395 3845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:41793,127.31.93.61:33657,127.31.93.60:41763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.61.keytab
--principal=kudu/127.31.93.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.61:33657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.93.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:35.508562 3845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:35.509790 3845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:35.520362 3850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:35.520412 3851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:35.520390 3856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:35.521255 3845 server_base.cc:1034] running on GCE node
I20260809 04:55:35.521907 3845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:35.523175 3845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:35.524387 3845 hybrid_clock.cc:648] HybridClock initialized: now 1786251335524321 us; error 41 us; skew 500 ppm
I20260809 04:55:35.524662 3845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:35 dist-test-slave-4tdj krb5kdc[3702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251335, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:35.527529 3845 init.cc:377] Logged in from keytab as kudu/127.31.93.61@KRBTEST.COM (short username kudu)
I20260809 04:55:35.528776 3845 webserver.cc:460] Webserver started at http://127.31.93.61:43483/ using document root <none> and password file <none>
I20260809 04:55:35.529374 3845 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:35.529471 3845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:35.529768 3845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:35.532541 3845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/data/instance:
uuid: "1d5d399bee3145758b0ab042c94cab35"
format_stamp: "Formatted at 2026-08-09 04:55:35 on dist-test-slave-4tdj"
server_key: "86befbc317a5e7c911c651c61d2cae2b"
server_key_iv: "a4ec40fb4300f922e13f4d60ecdca81d"
server_key_version: "encryptionkey@0"
I20260809 04:55:35.533273 3845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/wal/instance:
uuid: "1d5d399bee3145758b0ab042c94cab35"
format_stamp: "Formatted at 2026-08-09 04:55:35 on dist-test-slave-4tdj"
server_key: "86befbc317a5e7c911c651c61d2cae2b"
server_key_iv: "a4ec40fb4300f922e13f4d60ecdca81d"
server_key_version: "encryptionkey@0"
I20260809 04:55:35.537874 3845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 04:55:35.540977 3872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:35.542129 3845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 04:55:35.542332 3845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/wal
uuid: "1d5d399bee3145758b0ab042c94cab35"
format_stamp: "Formatted at 2026-08-09 04:55:35 on dist-test-slave-4tdj"
server_key: "86befbc317a5e7c911c651c61d2cae2b"
server_key_iv: "a4ec40fb4300f922e13f4d60ecdca81d"
server_key_version: "encryptionkey@0"
I20260809 04:55:35.542527 3845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:35.570057 3845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:35.573478 3845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:35.573715 3845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:35.589515 3845 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.61:33657
I20260809 04:55:35.589534 3952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.61:33657 every 8 connection(s)
I20260809 04:55:35.590797 3845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/data/info.pb
I20260809 04:55:35.594228 3954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:35.596846 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 3845
I20260809 04:55:35.597031 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-1/wal/instance
I20260809 04:55:35.597587 32116 external_mini_cluster.cc:1428] Setting key ac94d1e93d8fcde33bec7bec37068401
I20260809 04:55:35.602073 3954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:55:35 dist-test-slave-4tdj krb5kdc[3702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251335, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.61@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:55:35.621313 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.604889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33130 (local address 127.31.93.62:41793)
0809 04:55:35.605088 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:35.605094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:35.606287 (+ 1193us) server_negotiation.cc:407] Connection header received
0809 04:55:35.608086 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:35.608092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:35.608171 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:35.608281 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:35.609609 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:35.610522 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:35.611941 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:35.612349 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:35.612463 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:35.616048 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:35.616053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:35.616056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:35.616100 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:35.618546 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:35.619199 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:35.619205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:35.619207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:35.619326 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:35.619707 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:35.619712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:35.619715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:35.619977 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:35.620116 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:35.620766 (+ 650us) server_negotiation.cc:299] Negotiation successful
0809 04:55:35.620988 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 04:55:35.622153 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.605362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41793 (local address 127.0.0.1:33130)
0809 04:55:35.606103 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.606148 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:35.607826 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:35.608441 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:35.608448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:35.608927 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:35.609434 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:35.609451 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:35.610660 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:35.610664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:35.611793 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:35.611805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:35.612568 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:35.612576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:35.612842 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:35.613477 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:35.613507 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:35.615838 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:35.618671 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:35.618679 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:35.618694 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:35.619089 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:35.619427 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:35.619432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:35.619436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:35.619605 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:35.620096 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:35.620104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:35.620417 (+ 313us) client_negotiation.cc:769] Sending connection context
0809 04:55:35.620654 (+ 237us) client_negotiation.cc:240] Negotiation successful
0809 04:55:35.621111 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":314,"threads_started":1}
I20260809 04:55:35.623759 3954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.93.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.93.60@KRBTEST.COM" created.
Aug 09 04:55:35 dist-test-slave-4tdj krb5kdc[3702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251335, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.61@KRBTEST.COM for kudu/127.31.93.61@KRBTEST.COM
I20260809 04:55:35.637125 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.624364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:33657 (local address 127.0.0.1:54324)
0809 04:55:35.625140 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.625159 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:35.625270 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:35.625666 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:35.625670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:35.626068 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:35.626193 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:35.626201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:35.627495 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:35.627501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:35.628333 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:35.628344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:35.629073 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:35.629079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:35.629255 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:35.629700 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:35.629718 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:35.631956 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:35.634825 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:35.634830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:35.634833 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:35.635262 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:35.635574 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:35.635578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:35.635581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:35.635752 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:35.636275 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:35.636283 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:35.636413 (+ 130us) client_negotiation.cc:769] Sending connection context
0809 04:55:35.636544 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 04:55:35.636800 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679}
I20260809 04:55:35.637199 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.624577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54324 (local address 127.31.93.61:33657)
0809 04:55:35.625195 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:35.625203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:35.625244 (+ 41us) server_negotiation.cc:407] Connection header received
0809 04:55:35.625371 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:35.625378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:35.625538 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:35.625657 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:35.626365 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:35.627372 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:35.628452 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:35.628945 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:35.629050 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:35.632150 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:35.632160 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:35.632171 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:35.632203 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:35.634710 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:35.635370 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:35.635376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:35.635378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:35.635479 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:35.635850 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:35.635853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:35.635855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:35.636163 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:35.636316 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:35.636646 (+ 330us) server_negotiation.cc:299] Negotiation successful
0809 04:55:35.636901 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":451,"thread_start_us":318,"threads_started":1}
I20260809 04:55:35.639577 3954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:55:35.640617 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.640226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46614)
0809 04:55:35.640375 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.640496 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 04:55:35.640887 3877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
I20260809 04:55:35.642331 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.641976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46620)
0809 04:55:35.642121 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.642204 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 04:55:35.642580 3954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:41763: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.93.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.60.keytab.
Entry for principal kudu/127.31.93.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.60.keytab.
I20260809 04:55:35.661343 32116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
/tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.60:41763
--webserver_interface=127.31.93.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.60.keytab
--principal=kudu/127.31.93.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.93.20:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.93.62:41793,127.31.93.61:33657,127.31.93.60:41763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 04:55:35.691908 3954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } attempt: 1
I20260809 04:55:35.693078 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.692565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46624)
0809 04:55:35.692769 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.692888 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 04:55:35.694018 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.693656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46634)
0809 04:55:35.693803 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.693892 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 04:55:35.694219 3954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:41763: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
I20260809 04:55:35.762470 3954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } attempt: 2
I20260809 04:55:35.763669 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.763137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46640)
0809 04:55:35.763353 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.763470 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 04:55:35.764655 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.764288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46656)
0809 04:55:35.764427 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.764514 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 04:55:35.764860 3954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:41763: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
I20260809 04:55:35.795852 3829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } attempt: 5
Aug 09 04:55:35 dist-test-slave-4tdj krb5kdc[3702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251335, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for kudu/127.31.93.61@KRBTEST.COM
I20260809 04:55:35.809149 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.796576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54326 (local address 127.31.93.61:33657)
0809 04:55:35.796750 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:35.796756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:35.796843 (+ 87us) server_negotiation.cc:407] Connection header received
0809 04:55:35.797029 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:35.797033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:35.797114 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:35.797198 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:35.798009 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:35.798778 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:35.799920 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:35.800363 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:35.800475 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:35.804038 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:35.804043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:35.804046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:35.804079 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:35.806544 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:35.807374 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:35.807378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:35.807381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:35.807488 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:35.807901 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:35.807904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:35.807906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:35.808154 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:35.808258 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:35.808769 (+ 511us) server_negotiation.cc:299] Negotiation successful
0809 04:55:35.808937 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 04:55:35.809459 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.796547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:33657 (local address 127.0.0.1:54326)
0809 04:55:35.796749 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.796770 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:35.796904 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:35.797267 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:35.797273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:35.797668 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:35.797856 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:35.797865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:35.798908 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:35.798914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:35.799768 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:35.799781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:35.800502 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:35.800507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:35.800692 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:35.801351 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:35.801368 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:35.803849 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:35.806678 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:35.806685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:35.806689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:35.807254 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:35.807593 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:35.807598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:35.807602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:35.807796 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:35.808274 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:35.808280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:35.808469 (+ 189us) client_negotiation.cc:769] Sending connection context
0809 04:55:35.808705 (+ 236us) client_negotiation.cc:240] Negotiation successful
0809 04:55:35.809025 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 04:55:35.810561 3829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:55:35.811721 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.811244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46658)
0809 04:55:35.811434 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.811537 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 04:55:35.812729 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.812337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46670)
0809 04:55:35.812479 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.812580 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 04:55:35.812964 3829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:41763: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
I20260809 04:55:35.856089 3954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } attempt: 3
I20260809 04:55:35.857336 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.856749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46678)
0809 04:55:35.857002 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.857131 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260809 04:55:35.858443 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.857941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46682)
0809 04:55:35.858177 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.858269 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260809 04:55:35.858695 3954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:41763: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
I20260809 04:55:35.864159 3829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } attempt: 1
I20260809 04:55:35.865119 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.864735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46698)
0809 04:55:35.864883 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.864959 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 04:55:35.866073 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.865764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46712)
0809 04:55:35.865897 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.865966 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260809 04:55:35.866266 3829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:41763: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
I20260809 04:55:35.934476 3829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } attempt: 2
I20260809 04:55:35.935478 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.935040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46716)
0809 04:55:35.935228 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.935332 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 04:55:35.936444 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:35.936133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46730)
0809 04:55:35.936269 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:35.936332 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 04:55:35.936626 3829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:41763: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
W20260809 04:55:35.984866 3963 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 04:55:35.985188 3963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 04:55:35.985299 3963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 04:55:35.985414 3963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 04:55:35.992268 3963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 04:55:35.992355 3963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 04:55:35.992436 3963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 04:55:35.992499 3963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 04:55:35.992583 3963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 04:55:36.001857 3963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.93.20:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.93.62:41793,127.31.93.61:33657,127.31.93.60:41763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/krb5kdc/kudu_127.31.93.60.keytab
--principal=kudu/127.31.93.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.93.60:41763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.93.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:58:02 UTC on ccycloud.quasar-mitiho.root.comops.site
build id 4575794
ASAN enabled
I20260809 04:55:36.001929 3954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } attempt: 4
I20260809 04:55:36.003122 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.002566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46746)
0809 04:55:36.002820 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:36.002928 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 04:55:36.003358 3963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 04:55:36.004154 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.003801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46752)
0809 04:55:36.003946 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:36.004030 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 04:55:36.004355 3954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:41763: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
I20260809 04:55:36.004873 3963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 04:55:36.015461 3977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:36.015535 3981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 04:55:36.015578 3978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 04:55:36.016790 3963 server_base.cc:1034] running on GCE node
I20260809 04:55:36.017452 3963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 04:55:36.018702 3963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 04:55:36.019912 3963 hybrid_clock.cc:648] HybridClock initialized: now 1786251336019848 us; error 32 us; skew 500 ppm
I20260809 04:55:36.020201 3963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 04:55:36 dist-test-slave-4tdj krb5kdc[3702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251336, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 04:55:36.023098 3963 init.cc:377] Logged in from keytab as kudu/127.31.93.60@KRBTEST.COM (short username kudu)
I20260809 04:55:36.024367 3963 webserver.cc:460] Webserver started at http://127.31.93.60:45829/ using document root <none> and password file <none>
I20260809 04:55:36.024998 3963 fs_manager.cc:359] Metadata directory not provided
I20260809 04:55:36.025094 3963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 04:55:36.025398 3963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 04:55:36.028132 3963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/data/instance:
uuid: "bec959b58d194bb3920935419636431b"
format_stamp: "Formatted at 2026-08-09 04:55:36 on dist-test-slave-4tdj"
server_key: "b884286dcafe094cb5eb6abb6fcb69e7"
server_key_iv: "cab2d52c7b93fee7fba218c43467f20c"
server_key_version: "encryptionkey@0"
I20260809 04:55:36.028863 3963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/wal/instance:
uuid: "bec959b58d194bb3920935419636431b"
format_stamp: "Formatted at 2026-08-09 04:55:36 on dist-test-slave-4tdj"
server_key: "b884286dcafe094cb5eb6abb6fcb69e7"
server_key_iv: "cab2d52c7b93fee7fba218c43467f20c"
server_key_version: "encryptionkey@0"
I20260809 04:55:36.033528 3963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 04:55:36.036751 3992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 04:55:36.037957 3963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 04:55:36.038110 3963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/data,/tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/wal
uuid: "bec959b58d194bb3920935419636431b"
format_stamp: "Formatted at 2026-08-09 04:55:36 on dist-test-slave-4tdj"
server_key: "b884286dcafe094cb5eb6abb6fcb69e7"
server_key_iv: "cab2d52c7b93fee7fba218c43467f20c"
server_key_version: "encryptionkey@0"
I20260809 04:55:36.038313 3963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 04:55:36.042829 3829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } attempt: 3
I20260809 04:55:36.043900 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.043472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46758)
0809 04:55:36.043668 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:36.043747 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 04:55:36.044893 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.044478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46762)
0809 04:55:36.044705 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:36.044782 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260809 04:55:36.045089 3829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.93.60:41763: Network error: Client connection negotiation failed: client connection to 127.31.93.60:41763: connect: Connection refused (error 111)
I20260809 04:55:36.059587 3963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 04:55:36.063302 3963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 04:55:36.063529 3963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 04:55:36.079528 3963 rpc_server.cc:307] RPC server started. Bound to: 127.31.93.60:41763
I20260809 04:55:36.079547 4066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.93.60:41763 every 8 connection(s)
I20260809 04:55:36.080749 3963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/data/info.pb
I20260809 04:55:36.084187 4068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 04:55:36.084493 32116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu as pid 3963
I20260809 04:55:36.084654 32116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAAy6Yk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786250977734564-32116-0/minicluster-data/master-2/wal/instance
I20260809 04:55:36.085063 32116 external_mini_cluster.cc:1428] Setting key 92ae0247e0d423669fc1409145e143cd
I20260809 04:55:36.092542 4068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:55:36 dist-test-slave-4tdj krb5kdc[3702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.60@KRBTEST.COM
Aug 09 04:55:36 dist-test-slave-4tdj krb5kdc[3702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251336, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.60@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:55:36.109068 4072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.086698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.31.93.60:41763)
0809 04:55:36.087434 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.087444 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.087484 (+ 40us) server_negotiation.cc:407] Connection header received
0809 04:55:36.089669 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.089698 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.090061 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.090397 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.092024 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.093236 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.094900 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.095653 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.095826 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.100966 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:36.100976 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:36.100989 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:36.101029 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:36.105329 (+ 4300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.105944 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.105950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.105955 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.106094 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.106481 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.106484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.106487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.106914 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:36.107221 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.107583 (+ 362us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.107932 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":361,"threads_started":1}
I20260809 04:55:36.109475 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.095057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.62:41793 (local address 127.0.0.1:33144)
0809 04:55:36.095770 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:36.095808 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:36.096003 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:36.096514 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:36.096522 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:36.097064 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:36.097294 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:36.097302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.098664 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.098669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:36.100134 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:36.100148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.101151 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.101159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:36.101311 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.101881 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:36.101905 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:36.104201 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:36.107115 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:36.107125 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:36.107132 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:36.107517 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:36.107903 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:36.107907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:36.107910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:36.108058 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:36.108639 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:36.108644 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:36.108774 (+ 130us) client_negotiation.cc:769] Sending connection context
0809 04:55:36.108903 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 04:55:36.109125 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":363,"threads_started":1}
I20260809 04:55:36.109572 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.094994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.31.93.62:41793)
0809 04:55:36.095186 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.095193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.095886 (+ 693us) server_negotiation.cc:407] Connection header received
0809 04:55:36.096158 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.096164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.096264 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.096384 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.097438 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.098503 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.100301 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.100946 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.101052 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.104386 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:36.104390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:36.104394 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:36.104424 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:36.106987 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.107627 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.107632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.107635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.107765 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.108178 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.108183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.108187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.108520 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:36.108671 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.109031 (+ 360us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.109251 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:55:36.110973 4068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 04:55:36 dist-test-slave-4tdj krb5kdc[3702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251336, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.60@KRBTEST.COM for kudu/127.31.93.61@KRBTEST.COM
Aug 09 04:55:36 dist-test-slave-4tdj krb5kdc[3702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.62@KRBTEST.COM
I20260809 04:55:36.123359 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.111628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.31.93.61:33657)
0809 04:55:36.111804 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.111810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.111830 (+ 20us) server_negotiation.cc:407] Connection header received
0809 04:55:36.111923 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.111928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.112003 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.112079 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.112769 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.113415 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.114617 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.115024 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.115113 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.118339 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:36.118346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:36.118351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:36.118383 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:36.120927 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.121613 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.121618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.121622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.121774 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.122155 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.122158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.122160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.122435 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:36.122541 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.122923 (+ 382us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.123071 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 04:55:36.123377 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.111500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.61:33657 (local address 127.0.0.1:54336)
0809 04:55:36.111723 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:36.111740 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:36.111826 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:36.112133 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:36.112136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:36.112462 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:36.112632 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:36.112643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.113556 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.113560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:36.114487 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:36.114501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.115156 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.115162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:36.115349 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.115907 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:36.115928 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:36.118122 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:36.121046 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:36.121052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:36.121056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:36.121508 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:36.121872 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:36.121877 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:36.121881 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:36.122064 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:36.122544 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:36.122549 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:36.122694 (+ 145us) client_negotiation.cc:769] Sending connection context
0809 04:55:36.122851 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 04:55:36.123088 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260809 04:55:36.124243 4068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } has no permanent_uuid. Determining permanent_uuid...
I20260809 04:55:36.126051 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.111883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.31.93.62:41793)
0809 04:55:36.112242 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.112246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.112258 (+ 12us) server_negotiation.cc:407] Connection header received
0809 04:55:36.112291 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.112294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.112338 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.112404 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.113543 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.114208 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.115614 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.116107 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.116215 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.120229 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:36.120234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:36.120237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:36.120265 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:36.123125 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.123992 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.123997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.124000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.124149 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.124532 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.124536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.124538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.124852 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:36.125109 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.125546 (+ 437us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.125758 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
Aug 09 04:55:36 dist-test-slave-4tdj krb5kdc[3702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251336, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.60@KRBTEST.COM for kudu/127.31.93.60@KRBTEST.COM
Aug 09 04:55:36 dist-test-slave-4tdj krb5kdc[3702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.93.61@KRBTEST.COM
I20260809 04:55:36.136521 4072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.125018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.31.93.60:41763)
0809 04:55:36.125145 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.125149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.125268 (+ 119us) server_negotiation.cc:407] Connection header received
0809 04:55:36.125358 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.125363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.125430 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.125516 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.126649 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.127497 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.128609 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.129112 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.129219 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.131825 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:36.131829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:36.131832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:36.131861 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:36.134188 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.134752 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.134757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.134761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.134859 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.135220 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.135224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.135226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.135481 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:36.135620 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.135985 (+ 365us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.136282 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":36}
I20260809 04:55:36.136530 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.124917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46778)
0809 04:55:36.125170 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:36.125195 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:36.125286 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:36.125546 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:36.125550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:36.125969 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:36.126511 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:36.126531 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.127630 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.127636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:36.128489 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:36.128501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.129264 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.129270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:36.129486 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.129942 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:36.129957 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:36.131676 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:36.134329 (+ 2653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:36.134334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:36.134338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:36.134654 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:36.134951 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:36.134956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:36.134960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:36.135096 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:36.135592 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:36.135597 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:36.135757 (+ 160us) client_negotiation.cc:769] Sending connection context
0809 04:55:36.135895 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 04:55:36.136157 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260809 04:55:36.141844 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.127690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.31.93.61:33657)
0809 04:55:36.127842 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.127848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.128772 (+ 924us) server_negotiation.cc:407] Connection header received
0809 04:55:36.128854 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.128859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.128927 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.129021 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.130035 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.130756 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.131910 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.132516 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.132632 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.136598 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:36.136604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:36.136609 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:36.136644 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:36.139617 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.140164 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.140168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.140171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.140321 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.140616 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.140621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.140624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.140920 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:36.141222 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.141347 (+ 125us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.141554 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 04:55:36.142457 4068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b: Bootstrap starting.
I20260809 04:55:36.146183 4068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:36.146436 32116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 04:55:36.147303 4068 log.cc:824] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:36.149887 4068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b: No bootstrap required, opened a new log
I20260809 04:55:36.148727 3788 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33154
I20260809 04:55:36.156719 4068 raft_consensus.cc:348] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } }
I20260809 04:55:36.157004 4068 raft_consensus.cc:729] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bec959b58d194bb3920935419636431b, State: Initialized, Role: FOLLOWER
I20260809 04:55:36.157405 4068 consensus_queue.cc:260] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } }
I20260809 04:55:36.158254 4075 sys_catalog.cc:455] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } } }
I20260809 04:55:36.158658 4075 sys_catalog.cc:458] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:55:36.159271 4068 sys_catalog.cc:564] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b [sys.catalog]: configured and running, proceeding with master startup.
W20260809 04:55:36.167748 4100 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 04:55:36.167856 4100 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 04:55:36.166175 3963 master_runner.cc:186] Error getting master registration for 127.31.93.62:41793: OK, instance_id { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" instance_seqno: 1786251335092126 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 04:55:36.176749 3963 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 04:55:36.222329 3829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } attempt: 4
Aug 09 04:55:36 dist-test-slave-4tdj krb5kdc[3702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251335, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.62@KRBTEST.COM for kudu/127.31.93.60@KRBTEST.COM
I20260809 04:55:36.236419 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.223057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46784)
0809 04:55:36.223256 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:36.223278 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:36.223406 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:36.223769 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:36.223774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:36.224184 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:36.224345 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:36.224355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.225397 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.225403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:36.226361 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:36.226375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.227072 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.227078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:36.227276 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.227848 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:36.227866 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:36.230185 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:36.233652 (+ 3467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:36.233658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:36.233662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:36.234166 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:36.234536 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:36.234541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:36.234545 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:36.234721 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:36.235410 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:36.235416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:36.235600 (+ 184us) client_negotiation.cc:769] Sending connection context
0809 04:55:36.235748 (+ 148us) client_negotiation.cc:240] Negotiation successful
0809 04:55:36.236036 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 04:55:36.236438 4072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.223109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46784 (local address 127.31.93.60:41763)
0809 04:55:36.223298 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.223303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.223327 (+ 24us) server_negotiation.cc:407] Connection header received
0809 04:55:36.223539 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.223544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.223631 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.223732 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.224493 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.225256 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.226493 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.226939 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.227046 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.230392 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:36.230400 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:36.230405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:36.230443 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:36.233532 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.234275 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.234280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.234284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.234435 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.234821 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.234826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.234830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.235271 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:36.235424 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.235863 (+ 439us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.236083 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 04:55:36.240886 3829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b: Bootstrap starting.
I20260809 04:55:36.244766 3829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:36.246016 3829 log.cc:824] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:36.249239 3829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b: No bootstrap required, opened a new log
I20260809 04:55:36.256399 3829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } }
I20260809 04:55:36.256764 3829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a970f39d7f947fea1ffb0ec9d146a5b, State: Initialized, Role: FOLLOWER
I20260809 04:55:36.257244 3829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } }
I20260809 04:55:36.258399 4103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } } }
I20260809 04:55:36.258986 4103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:55:36.259488 3829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:36.259599 3788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 04:55:36.259775 3788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:36.263175 3788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } }
I20260809 04:55:36.264933 3788 leader_election.cc:288] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [CANDIDATE]: Term 1 election: Requested vote from peers 1d5d399bee3145758b0ab042c94cab35 (127.31.93.61:33657), bec959b58d194bb3920935419636431b (127.31.93.60:41763)
I20260809 04:55:36.266206 4032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bec959b58d194bb3920935419636431b"
I20260809 04:55:36.266690 4032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:36.270066 4032 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a970f39d7f947fea1ffb0ec9d146a5b in term 1.
I20260809 04:55:36.270850 3757 leader_election.cc:302] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a970f39d7f947fea1ffb0ec9d146a5b, bec959b58d194bb3920935419636431b; no voters:
I20260809 04:55:36.271313 4103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 04:55:36.284029 4103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [term 1 LEADER]: Becoming Leader. State: Replica: 5a970f39d7f947fea1ffb0ec9d146a5b, State: Running, Role: LEADER
I20260809 04:55:36.285050 4103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } }
I20260809 04:55:36.294806 4117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a970f39d7f947fea1ffb0ec9d146a5b. Latest consensus state: current_term: 1 leader_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } } }
I20260809 04:55:36.295591 4117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:36.302068 4126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 04:55:36.303560 3954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } attempt: 5
W20260809 04:55:36.315502 4001 tablet.cc:2369] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 04:55:36.320874 4126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260809 04:55:36.332805 3759 tablet.cc:2369] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 04:55:36 dist-test-slave-4tdj krb5kdc[3702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786251335, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.93.61@KRBTEST.COM for kudu/127.31.93.60@KRBTEST.COM
I20260809 04:55:36.350657 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.301650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33158 (local address 127.31.93.62:41793)
0809 04:55:36.301818 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.301824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.305273 (+ 3449us) server_negotiation.cc:407] Connection header received
0809 04:55:36.306323 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.306330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.306417 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.306489 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.316485 (+ 9996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.317658 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.322322 (+ 4664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.322987 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.323087 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.336808 (+ 13721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:36.336816 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:36.336821 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:36.336867 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:36.340560 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.345978 (+ 5418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.345984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.345989 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.346196 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.347442 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.347447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.347450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.347816 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:36.348070 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.349995 (+ 1925us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.350277 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 04:55:36.352593 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.304152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.93.60:41763 (local address 127.0.0.1:46798)
0809 04:55:36.304498 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0809 04:55:36.304519 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 04:55:36.304638 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 04:55:36.325644 (+ 21006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 04:55:36.325649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 04:55:36.326198 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0809 04:55:36.326420 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:36.326433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.330407 (+ 3974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.330413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:36.331881 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 04:55:36.331896 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.341764 (+ 9868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.341772 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 04:55:36.342020 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.342569 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 04:55:36.342593 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 04:55:36.345705 (+ 3112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 04:55:36.349724 (+ 4019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:36.349730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:36.349734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:36.350324 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:36.350781 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 04:55:36.350787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 04:55:36.350791 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 04:55:36.350961 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 04:55:36.351583 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 04:55:36.351588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 04:55:36.351746 (+ 158us) client_negotiation.cc:769] Sending connection context
0809 04:55:36.351877 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 04:55:36.352179 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226}
I20260809 04:55:36.353783 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.304836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.31.93.60:41763)
0809 04:55:36.325181 (+ 20345us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.325187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.325214 (+ 27us) server_negotiation.cc:407] Connection header received
0809 04:55:36.325284 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.325289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.325369 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.326579 (+ 1210us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.326838 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.328039 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.332100 (+ 4061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.333767 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.342364 (+ 8597us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.346015 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:36.346022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:36.346027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:36.346058 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:36.349565 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.350490 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.350496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.350500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.350653 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.351111 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.351116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.351120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.351444 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:36.352978 (+ 1534us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.353135 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.353367 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20188,"thread_start_us":20624,"threads_started":1}
I20260809 04:55:36.357468 4072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.302648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.31.93.60:41763)
0809 04:55:36.302809 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.302814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.309292 (+ 6478us) server_negotiation.cc:407] Connection header received
0809 04:55:36.310473 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.310479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.310561 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.310628 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.319490 (+ 8862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.320552 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.329351 (+ 8799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.330035 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.330132 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.341444 (+ 11312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:36.341451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:36.341456 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:36.341499 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:36.345239 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.347098 (+ 1859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.347103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.347107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.347279 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.348832 (+ 1553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.348837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.348840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.349186 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:36.349451 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.356926 (+ 7475us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.357144 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 04:55:36.359956 3954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35: Bootstrap starting.
I20260809 04:55:36.362437 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.309746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.31.93.61:33657)
0809 04:55:36.309913 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.309919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.313565 (+ 3646us) server_negotiation.cc:407] Connection header received
0809 04:55:36.314175 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.314194 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.314333 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.314415 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.324064 (+ 9649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.325111 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.333040 (+ 7929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.333670 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.333766 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.352616 (+ 18850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 04:55:36.352622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 04:55:36.352626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 04:55:36.352664 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 04:55:36.356038 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.359854 (+ 3816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.359860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.359862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.360043 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 04:55:36.360503 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 04:55:36.360509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 04:55:36.360512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 04:55:36.360869 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 04:55:36.361093 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.361646 (+ 553us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.361869 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 04:55:36.362497 4032 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b [term 1 FOLLOWER]: Refusing update from remote peer 5a970f39d7f947fea1ffb0ec9d146a5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:55:36.363941 4117 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:55:36.371446 3954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35: Neither blocks nor log segments found. Creating new log.
I20260809 04:55:36.375731 3954 log.cc:824] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35: Log is configured to *not* fsync() on all Append() calls
I20260809 04:55:36.383985 4075 sys_catalog.cc:455] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a970f39d7f947fea1ffb0ec9d146a5b. Latest consensus state: current_term: 1 leader_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } } }
I20260809 04:55:36.384289 4075 sys_catalog.cc:458] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:55:36.384922 3954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35: No bootstrap required, opened a new log
I20260809 04:55:36.385673 4140 mvcc.cc:205] Tried to move back new op lower bound from 7316485473717850112 to 7316485473433436160. Current Snapshot: MvccSnapshot[applied={T|T < 7316485473717850112}]
I20260809 04:55:36.375188 3728 master_runner.cc:186] Error getting master registration for 127.31.93.61:33657: OK, instance_id { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" instance_seqno: 1786251335585126 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 04:55:36.395144 3728 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 04:55:36.397085 3954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } }
I20260809 04:55:36.397753 3954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5d399bee3145758b0ab042c94cab35, State: Initialized, Role: FOLLOWER
I20260809 04:55:36.398627 3954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } }
I20260809 04:55:36.399096 4117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } } }
I20260809 04:55:36.399366 4117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:36.405825 4154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } } }
I20260809 04:55:36.407168 4154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:55:36.408308 3909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1d5d399bee3145758b0ab042c94cab35"
I20260809 04:55:36.408671 4126 catalog_manager.cc:1347] Generated new cluster ID: a98412e7c19f4aa6bbd7119311a79d98
I20260809 04:55:36.408778 4126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 04:55:36.408739 3909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35 [term 0 FOLLOWER]: Advancing to term 1
I20260809 04:55:36.411917 3909 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a970f39d7f947fea1ffb0ec9d146a5b in term 1.
I20260809 04:55:36.412801 3908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35 [term 1 FOLLOWER]: Refusing update from remote peer 5a970f39d7f947fea1ffb0ec9d146a5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 04:55:36.413327 3954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 04:55:36.413635 4103 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 04:55:36.426235 4075 sys_catalog.cc:455] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } } }
I20260809 04:55:36.426517 4075 sys_catalog.cc:458] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:55:36.440784 4126 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 04:55:36.443317 4126 catalog_manager.cc:1504] Loading token signing keys...
I20260809 04:55:36.456557 4117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } } }
I20260809 04:55:36.456822 4117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b [sys.catalog]: This master's current role is: LEADER
I20260809 04:55:36.458215 4154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a970f39d7f947fea1ffb0ec9d146a5b. Latest consensus state: current_term: 1 leader_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } } }
I20260809 04:55:36.458566 4154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:55:36.458899 4163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a970f39d7f947fea1ffb0ec9d146a5b" member_type: VOTER last_known_addr { host: "127.31.93.62" port: 41793 } } peers { permanent_uuid: "1d5d399bee3145758b0ab042c94cab35" member_type: VOTER last_known_addr { host: "127.31.93.61" port: 33657 } } peers { permanent_uuid: "bec959b58d194bb3920935419636431b" member_type: VOTER last_known_addr { host: "127.31.93.60" port: 41763 } } }
I20260809 04:55:36.459316 4163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 04:55:36.490350 4126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a970f39d7f947fea1ffb0ec9d146a5b: Generated new TSK 0
I20260809 04:55:36.491330 4126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 04:55:36.525025 4181 catalog_manager.cc:1259] Loaded cluster ID: a98412e7c19f4aa6bbd7119311a79d98
I20260809 04:55:36.525128 4181 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35: loading cluster ID for follower catalog manager: success
I20260809 04:55:36.529255 4181 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35: acquiring CA information for follower catalog manager: success
I20260809 04:55:36.531657 4181 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d5d399bee3145758b0ab042c94cab35: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 04:55:36.627345 4132 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:36.615953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.31.93.60:41763)
0809 04:55:36.616147 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.616154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.625457 (+ 9303us) server_negotiation.cc:407] Connection header received
0809 04:55:36.625670 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.625675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.625774 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.625837 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.627097 (+ 1260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46808: BlockingRecv error: recv got EOF from 127.0.0.1:46808 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 04:55:36.628983 3832 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:36.612173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.31.93.62:41793)
0809 04:55:36.612381 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.612387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.623789 (+ 11402us) server_negotiation.cc:407] Connection header received
0809 04:55:36.625575 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.625582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.625695 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.625769 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.628737 (+ 2968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33162: BlockingRecv error: recv got EOF from 127.0.0.1:33162 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260809 04:55:36.629963 3959 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:36.624299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54374 (local address 127.31.93.61:33657)
0809 04:55:36.624478 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.624483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.627992 (+ 3509us) server_negotiation.cc:407] Connection header received
0809 04:55:36.628527 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.628533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.628633 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.628699 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.629744 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54374: BlockingRecv error: recv got EOF from 127.0.0.1:54374 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 04:55:36.674335 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.666247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54386 (local address 127.31.93.61:33657)
0809 04:55:36.666502 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.666508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.667003 (+ 495us) server_negotiation.cc:407] Connection header received
0809 04:55:36.667315 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.667321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.667502 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.667638 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:36.668328 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.669452 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.671434 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.672094 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.672195 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.672876 (+ 681us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:36.673541 (+ 665us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:36.673662 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.673904 (+ 242us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.673996 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
W20260809 04:55:36.715363 4132 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:36.706317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.31.93.60:41763)
0809 04:55:36.706500 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.706507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.713313 (+ 6806us) server_negotiation.cc:407] Connection header received
0809 04:55:36.713819 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.713826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.713956 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.714033 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.715076 (+ 1043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46820: BlockingRecv error: recv got EOF from 127.0.0.1:46820 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 04:55:36.758255 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.749848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.31.93.61:33657)
0809 04:55:36.750064 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.750071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.752235 (+ 2164us) server_negotiation.cc:407] Connection header received
0809 04:55:36.752425 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.752430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.752529 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.752622 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:36.753109 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.754234 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.756148 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.756851 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.756984 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.757480 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:36.757625 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:36.757712 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.757865 (+ 153us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.757947 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260809 04:55:36.768448 4132 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:36.765859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.31.93.60:41763)
0809 04:55:36.766051 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.766057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.766874 (+ 817us) server_negotiation.cc:407] Connection header received
0809 04:55:36.767168 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.767174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.767272 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.767383 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.768221 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46832: BlockingRecv error: recv got EOF from 127.0.0.1:46832 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 04:55:36.779870 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.773228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.31.93.61:33657)
0809 04:55:36.773403 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.773408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.773917 (+ 509us) server_negotiation.cc:407] Connection header received
0809 04:55:36.774076 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.774082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.774181 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.774275 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:36.774752 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.775869 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.777781 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.778494 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.778634 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.779071 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:36.779216 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:36.779315 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.779471 (+ 156us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.779551 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260809 04:55:36.788160 4132 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:36.786142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.31.93.60:41763)
0809 04:55:36.786358 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.786364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.786743 (+ 379us) server_negotiation.cc:407] Connection header received
0809 04:55:36.786926 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.786931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.787031 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.787150 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.787937 (+ 787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46848: BlockingRecv error: recv got EOF from 127.0.0.1:46848 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260809 04:55:36.801010 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.794665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.31.93.61:33657)
0809 04:55:36.794828 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.794833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.795293 (+ 460us) server_negotiation.cc:407] Connection header received
0809 04:55:36.795417 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.795423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.795499 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.795596 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:36.795984 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.797070 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.798983 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.799677 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.799803 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.800254 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:36.800399 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:36.800497 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.800658 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.800737 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260809 04:55:36.809734 4132 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:36.807440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46860 (local address 127.31.93.60:41763)
0809 04:55:36.807726 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.807731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.808299 (+ 568us) server_negotiation.cc:407] Connection header received
0809 04:55:36.808482 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.808487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.808576 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.808673 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.809497 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46860: BlockingRecv error: recv got EOF from 127.0.0.1:46860 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
I20260809 04:55:36.827490 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.820671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.31.93.61:33657)
0809 04:55:36.820859 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.820865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.821349 (+ 484us) server_negotiation.cc:407] Connection header received
0809 04:55:36.821565 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.821571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.821658 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.821747 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:36.822236 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.823411 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.825355 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.826040 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.826190 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.826664 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:36.826818 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:36.826913 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.827087 (+ 174us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.827166 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260809 04:55:36.838548 4132 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:36.836381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.31.93.60:41763)
0809 04:55:36.836556 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.836562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.837094 (+ 532us) server_negotiation.cc:407] Connection header received
0809 04:55:36.837253 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.837258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.837349 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.837459 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.838328 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46876: BlockingRecv error: recv got EOF from 127.0.0.1:46876 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 04:55:36.865485 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.857578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.31.93.61:33657)
0809 04:55:36.857769 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.857775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.858575 (+ 800us) server_negotiation.cc:407] Connection header received
0809 04:55:36.858787 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.858793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.858913 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.859014 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:36.859709 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.860971 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.863134 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.863778 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.863906 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.864422 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:36.864557 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:36.864647 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.864889 (+ 242us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.864986 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260809 04:55:36.875056 4132 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:36.872623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46890 (local address 127.31.93.60:41763)
0809 04:55:36.872822 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.872828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.873441 (+ 613us) server_negotiation.cc:407] Connection header received
0809 04:55:36.873624 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.873630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.873723 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.873828 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.874785 (+ 957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46890: BlockingRecv error: recv got EOF from 127.0.0.1:46890 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 04:55:36.922817 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:36.915892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.31.93.61:33657)
0809 04:55:36.916112 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.916119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.916632 (+ 513us) server_negotiation.cc:407] Connection header received
0809 04:55:36.916826 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.916832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.916974 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.917102 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:36.917578 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.918747 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.920660 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:36.921367 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:36.921503 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:36.921959 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:36.922112 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:36.922223 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:36.922394 (+ 171us) server_negotiation.cc:299] Negotiation successful
0809 04:55:36.922479 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260809 04:55:36.933271 4132 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:36.931530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.31.93.60:41763)
0809 04:55:36.931714 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:36.931720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:36.931970 (+ 250us) server_negotiation.cc:407] Connection header received
0809 04:55:36.932154 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:36.932158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:36.932231 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:36.932309 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:36.933052 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46896: BlockingRecv error: recv got EOF from 127.0.0.1:46896 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 04:55:37.005344 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:37.000433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.31.93.61:33657)
0809 04:55:37.000621 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:37.000628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:37.000990 (+ 362us) server_negotiation.cc:407] Connection header received
0809 04:55:37.001200 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:37.001204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:37.001271 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:37.001355 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:37.001759 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:37.002516 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:37.003820 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:37.004197 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:37.004305 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:37.004685 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:37.004786 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:37.004858 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:37.004995 (+ 137us) server_negotiation.cc:299] Negotiation successful
0809 04:55:37.005069 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260809 04:55:37.011754 4132 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:37.010231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46908 (local address 127.31.93.60:41763)
0809 04:55:37.010462 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:37.010467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:37.010682 (+ 215us) server_negotiation.cc:407] Connection header received
0809 04:55:37.010826 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:37.010829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:37.010902 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:37.010982 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:37.011544 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46908: BlockingRecv error: recv got EOF from 127.0.0.1:46908 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260809 04:55:37.148169 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:37.143377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.31.93.61:33657)
0809 04:55:37.143597 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:37.143603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:37.143858 (+ 255us) server_negotiation.cc:407] Connection header received
0809 04:55:37.144055 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:37.144059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:37.144138 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:37.144216 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:37.144635 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:37.145345 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:37.146646 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:37.147057 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:37.147163 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:37.147539 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:37.147641 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:37.147723 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:37.147850 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 04:55:37.147926 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260809 04:55:37.155297 4132 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 04:55:37.153501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.31.93.60:41763)
0809 04:55:37.153719 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:37.153725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:37.153916 (+ 191us) server_negotiation.cc:407] Connection header received
0809 04:55:37.154078 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:37.154083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:37.154153 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:37.154238 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 04:55:37.155054 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46920: BlockingRecv error: recv got EOF from 127.0.0.1:46920 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 04:55:37.169610 4100 catalog_manager.cc:1259] Loaded cluster ID: a98412e7c19f4aa6bbd7119311a79d98
I20260809 04:55:37.169689 4100 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b: loading cluster ID for follower catalog manager: success
I20260809 04:55:37.172022 4100 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b: acquiring CA information for follower catalog manager: success
I20260809 04:55:37.173666 4100 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bec959b58d194bb3920935419636431b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 04:55:37.419812 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:37.415388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.31.93.61:33657)
0809 04:55:37.415614 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:37.415619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:37.415848 (+ 229us) server_negotiation.cc:407] Connection header received
0809 04:55:37.416005 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:37.416009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:37.416081 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:37.416159 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:37.416502 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:37.417216 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:37.418492 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:37.418877 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:37.418960 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:37.419262 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:37.419364 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:37.419429 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:37.419506 (+ 77us) server_negotiation.cc:299] Negotiation successful
0809 04:55:37.419564 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 04:55:37.429792 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:37.425081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.31.93.60:41763)
0809 04:55:37.425252 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:37.425257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:37.425445 (+ 188us) server_negotiation.cc:407] Connection header received
0809 04:55:37.425589 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:37.425593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:37.425744 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:37.425836 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:37.426162 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:37.426867 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:37.428029 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:37.428411 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:37.428496 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:37.428884 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:37.429396 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:37.429498 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:37.429535 (+ 37us) server_negotiation.cc:299] Negotiation successful
0809 04:55:37.429607 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 04:55:37.443933 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:37.437582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54460 (local address 127.31.93.61:33657)
0809 04:55:37.437754 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:37.437760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:37.438542 (+ 782us) server_negotiation.cc:407] Connection header received
0809 04:55:37.438701 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:37.438707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:37.438783 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:37.438861 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:37.439497 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:37.440286 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:37.442007 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:37.442531 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:37.442647 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:37.443149 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:37.443245 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:37.443324 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:37.443523 (+ 199us) server_negotiation.cc:299] Negotiation successful
0809 04:55:37.443608 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 04:55:37.444098 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:37.435045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.31.93.60:41763)
0809 04:55:37.435208 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:37.435213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:37.438135 (+ 2922us) server_negotiation.cc:407] Connection header received
0809 04:55:37.438374 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:37.438379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:37.438455 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:37.438540 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:37.439036 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:37.439802 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:37.442078 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:37.442717 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:37.442816 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:37.443362 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:37.443495 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:37.443567 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:37.443759 (+ 192us) server_negotiation.cc:299] Negotiation successful
0809 04:55:37.443831 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 04:55:37.444553 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 04:55:37.435054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33172 (local address 127.31.93.62:41793)
0809 04:55:37.436802 (+ 1748us) server_negotiation.cc:206] Beginning negotiation
0809 04:55:37.436809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 04:55:37.437498 (+ 689us) server_negotiation.cc:407] Connection header received
0809 04:55:37.437751 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 04:55:37.437757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 04:55:37.437950 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 04:55:37.438120 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 04:55:37.438764 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:37.439701 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:37.441813 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 04:55:37.442494 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 04:55:37.442629 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 04:55:37.443166 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 04:55:37.443550 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 04:55:37.443646 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 04:55:37.444223 (+ 577us) server_negotiation.cc:299] Negotiation successful
0809 04:55:37.444293 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1606,"thread_start_us":1854,"threads_started":1}
I20260809 04:55:37.448676 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 3728
I20260809 04:55:37.449931 3821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:55:37.556231 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 3728
W20260809 04:55:37.583837 3997 connection.cc:535] server connection from 127.0.0.1:46784 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 04:55:37.583869 3876 connection.cc:535] server connection from 127.0.0.1:54326 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 04:55:37.584244 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 3845
I20260809 04:55:37.585207 3947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:55:37.689767 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 3845
I20260809 04:55:37.714208 32116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu pid 3963
I20260809 04:55:37.715322 4060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 04:55:37.818228 32116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAAy6Yk/build/asan/bin/kudu with pid 3963
2026-08-09T04:55:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4601 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9033 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (359588 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST